{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fe98a9e5-4646-47d0-9445-19bf22ef3a54",
   "metadata": {},
   "source": [
    "# <center>Ch2 Lora微调原理详解 </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74361e71",
   "metadata": {},
   "source": [
    "# 1.为什么需要Lora"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e185a65",
   "metadata": {},
   "source": [
    "1. 全参微调的难点：\n",
    "- 大型语言模型的微调成本高昂：完整微调需要存储和更新所有模型参数的副本\n",
    "- 部署困难：每个任务都需要存储完整的模型副本,占用大量存储空间\n",
    "- 计算资源要求高：完整微调需要大量计算资源和GPU内存"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "471754f3",
   "metadata": {},
   "source": [
    "2. LoRA的核心思想：\n",
    "- 假设模型权重的更新可以通过低秩分解来表示\n",
    "- 不直接更新原始权重矩阵,而是训练两个更小的矩阵(低秩分解)\n",
    "- 通过这种方式大大减少了需要训练和存储的参数数量"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e86b303",
   "metadata": {},
   "source": [
    "3. LoRA的优势：\n",
    "- 显著减少可训练参数数量\n",
    "- 降低GPU内存需求\n",
    "- 加快训练速度\n",
    "- 多个任务可以共享基础模型,只需存储小型任务特定适配器\n",
    "- 训练稳定性好,性能接近完整微调"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eed09fe",
   "metadata": {},
   "source": [
    "假设一个在线客服场景:\n",
    "- 公司有一个基础的大语言模型用于客服对话需要为不同产品线(如手机、电脑、家电等)定制专门的客服机器人，传统方法需要为每个产品线存储一个完整模型副本\n",
    "- 使用LoRA后:\n",
    "  - 只需维护一个基础模型\n",
    "  - 为每个产品线训练小型LoRA适配器\n",
    "  - 运行时动态加载对应产品线的适配器\n",
    "  - 大大节省存储空间和训练成本\n",
    "  - 方便快速增加新产品线的支持\n",
    "\n",
    "这样不仅降低了部署和维护成本,还提高了模型更新和扩展的灵活性。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "427f1a24",
   "metadata": {},
   "source": [
    "# 2.Lora微调方法讲解"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6bbf7f9",
   "metadata": {},
   "source": [
    "## 2.1 Lora的介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4af9818e",
   "metadata": {},
   "source": [
    "&emsp;&emsp;LoRA来源于微软在2021年发布的Paper：《LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS》 低秩矩阵微调<br>\n",
    "&emsp;&emsp;地址：https://arxiv.org/pdf/2106.09685.pdf ， 同时也在Github上开源了LoRA的技术实现过程，其Github地址：https://github.com/microsoft/LoRA 。\n",
    "```json\n",
    "    适配器微调（Adapter Tuning）是在模型中引入了计算模块，本质还是要在模型本身增加层次，会让模型推理走更长的路径。   \n",
    "    LORA的核心思想基准模型不进行变化，额外引入一部分参数来做专属内容处理，同时加上原有模型的推理能力，这部分新增加的的内容就是要训练出来的参数矩阵。\n",
    "```\n",
    "新的矩阵 $ h $ 可以表示为：\n",
    "\n",
    "$ h = W_0 x + \\Delta W x $\n",
    "\n",
    "其中：\n",
    "- $ W_0 $ 是原始的权重矩阵。\n",
    "- $ x $ 是输入向量。\n",
    "- $ \\Delta W $ 是新增的权重矩阵，用于增加训练的特色。\n",
    "\n",
    " $ x $ 经过原有矩阵计算和现在新增$ \\Delta W $ 是矩阵计算后权重给出一个期望的值。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61d9debe",
   "metadata": {},
   "source": [
    "<div align=center><img src=\"https://typora-photo1220.oss-cn-beijing.aliyuncs.com/DataAnalysis/muyan/image-20241110165413671.png\" width=90%></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33131936",
   "metadata": {},
   "source": [
    "&emsp;&emsp;新增这部分参数矩阵我期望不要太大，不然和全参数微调没什么区别了，我想参数量小一点，节约点资源。不然就成了全参数微调了！\n",
    "线性代数上讲一个参数矩阵可以由两个矩阵相乘得出来，同理$ \\Delta W $ 就可以将本身的矩阵拆成矩阵A和矩阵B，A*B=$ \\Delta W $ <br>\n",
    "原有算法 $ h = W_0 x + \\Delta W x $ 变成 $h = W_0x + \\Delta Wx = W_0x + BAx$    ( $ x $ 做为向量入参不需要改变只需要跟随即可)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1752b2f4",
   "metadata": {},
   "source": [
    "这里引入了一个概念\n",
    "> 本征维度（Intrinsic Dimension）：是指数据或空间中所需的最小维度，以便充分描述其中的结构或特征。换句话说，尽管数据可能存在于高维空间中，但其实际所包含的信息可能集中在一个更低维度的子空间内，本征维度就是描述这个低维子空间的维度。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "710dbd94",
   "metadata": {},
   "source": [
    "1. LoRA与本征维度的关系：\n",
    "- LoRA的核心假设是神经网络的权重更新矩阵通常具有较低的\"本征秩\"\n",
    "- 这意味着虽然模型权重是高维的,但实际的任务相关更新可能位于一个低维子空间中\n",
    "- LoRA正是利用了这一特性,使用低秩分解来捕获权重更新\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfe6b9d2",
   "metadata": {},
   "source": [
    "\n",
    "<div align=center>\n",
    "        <td><img src=\"https://typora-photo1220.oss-cn-beijing.aliyuncs.com/DataAnalysis/muyan/image-20241117150602234.png\" width=20%></td>\n",
    "        <td><img src=\"https://typora-photo1220.oss-cn-beijing.aliyuncs.com/DataAnalysis/muyan/image-20241117150516158.png\" width=20%></td>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c54b4bed",
   "metadata": {},
   "source": [
    "```json\n",
    "    例如一张苹果图片，展示很精细，有光线，斑点、叶子脉络、细微的色差等等。\n",
    "    但是我想要的就是能让我看出苹果才是最关键的点，所以上边的描述是没用的，仅仅用右边最普通的像素点展示就可以了。这个像素极低的几个点才是我最关键的信息。\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc555fc",
   "metadata": {},
   "source": [
    "&emsp;&emsp;同理，引入到矩阵中如果将$ \\Delta W $ 拆成矩阵A和矩阵B，这两个矩阵的秩与原矩阵相同那就没有意义了，因为想要训练某一个方向专属技能就代表其他技能的参数在想要输出的方向上没那么重要，是不是提炼最相关的那部分业务数据相对更重要。<br>\n",
    "就会有个关键信息，低秩矩阵，低秩是什么？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b25035be",
   "metadata": {},
   "source": [
    "<div align=center><img src=\"https://typora-photo1220.oss-cn-beijing.aliyuncs.com/DataAnalysis/muyan/image-20241117154303169.png\" width=100%></div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "405061a6",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{bmatrix}\n",
    "2*3 & 2*2 & 10 \\\\\n",
    "1*3 & 3*2 & 9 \\\\\n",
    "3*3 & 1*2 & 11\n",
    "\\end{bmatrix}\n",
    "\\quad\n",
    "\\begin{bmatrix}\n",
    "2*3 & 1*2 & 8 \\\\\n",
    "1*3 & 1*2 & 5 \\\\\n",
    "\\end{bmatrix}\n",
    "\\quad\n",
    "\\begin{bmatrix}\n",
    "1*3 & 1*2 & 5 \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6990e2c1",
   "metadata": {},
   "source": [
    "一个矩阵的秩是指，矩阵中线性独立行或列的最大数目<br>\n",
    "第一个矩阵，A、B、C分别各自表示花了多少钱，谁也不能替代谁<br>\n",
    "第二个矩阵，C买的东西花的钱是A和B的和，那么C就可以被替代，就不属于秩的概念<br>\n",
    "第三个矩阵，C买的东西花的钱是A*3的倍数，那么C就不用表示，B买的东西花的钱是A*2的倍数，那么B也就不用表示<br>\n",
    "对应的秩(Rank) 分别为 3 、2 、1 ，我们是不是可以理解同秩越低占用空间越小<br>\n",
    "忽略严谨的数学概念定义就可以理解一个矩阵的秩越大，信息含量就越大，对应占用空间也就越大。也就是为什么要用低秩！<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8746486b",
   "metadata": {},
   "source": [
    "具体体现：\n",
    "```python\n",
    "    # 假设原始权重矩阵 W ∈ R^{d×k}\n",
    "    # LoRA将权重更新分解为:\n",
    "    ΔW = BA   # 其中B ∈ R^{d×r}, A ∈ R^{r×k}, r << min(d,k)\n",
    "```\n",
    "- r 是LoRA的秩(rank),通常远小于原始维度\n",
    "- r 实际上反映了任务适配所需的本征维度\n",
    "- 较小的r就足够表达任务所需的权重更新,说明更新确实存在于低维子空间"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53b780d4",
   "metadata": {},
   "source": [
    "矩阵是什么呢？怎么表示？<br>\n",
    "小明想去三个地方距离分别是10、20、30km<br>\n",
    "他可以步行、自行车、骑摩托去 分别花费了多少时间。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3c2a568",
   "metadata": {},
   "source": [
    "<div align=center><img src=\"https://typora-photo1220.oss-cn-beijing.aliyuncs.com/DataAnalysis/muyan/image-20241117161939882.png\" width=50%></div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d491b407",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{bmatrix}\n",
    "@ \\\\\n",
    "@ \\\\\n",
    "@ \\\\\n",
    "\\end{bmatrix}\n",
    "与\n",
    "\\begin{bmatrix}\n",
    "@ & @ & @ \\\\\n",
    "\\end{bmatrix}\n",
    "能够表达\n",
    "\\begin{bmatrix}\n",
    "@ & @ & @ \\\\\n",
    "@ & @ & @ \\\\\n",
    "@ & @ & @\n",
    "\\end{bmatrix}\n",
    "矩阵的数量\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "480c3894",
   "metadata": {},
   "source": [
    "```json\n",
    "    @ 符号仅作为数字占位符使用，不做实际意义，仅用来表达两组数据可以表达一个n*n的矩阵\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ddd8cb0",
   "metadata": {},
   "source": [
    "相比于一开始矩阵9个参数当前替换后的两个小矩阵参数就是3+3=6<br>\n",
    "```json\n",
    "    对比之前就是9-6 少了3个参数量的位置与空间。\n",
    "    如果是一个100w的矩阵的规模，就可以拆成1000*1000的两个矩阵，假设秩=8 那么参数量就是1000*8+1000*8=16000\n",
    "    这个参数量占原有100w惨数量的 16000/1000000=1.6%\n",
    "    对于动辄10亿的参数节约的空间就很大了\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef643ff8",
   "metadata": {},
   "source": [
    " > 过参数化模型（Over-parametrized models）是指参数数量远超训练数据样本数量的模型。换句话说，模型中可调参数的数量大于用于训练的样本数量。如GPT系列模型和transformer模型均为此类模型。采用这种策略可以带来性能上的大幅提升。\n",
    "\n",
    "&emsp;&emsp;同样的，如此大的参数量，一定是有冗余的，也就是说：如此多的参数对于很多下游任务，不一定都是有用的。垂直领域的知识所需要的参数一定远小于全量参数，对这样的任务，可能仅仅一部分重要的参数就可以做的足够好。实际上存在于一个较低的内在维度上。   \n",
    " - 假设模型适应过程中权重的变化也具有较低的“内在秩”，基于此这导致了低秩适应（LoRA）方法的灵感。LoRA允许我们通过优化适应过程中密集层变化的秩分解矩阵来间接训练神经网络中的一些密集层，同时保持预先训练的权值冻结，以实现微调效果并节省了计算资源。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b5b84ea",
   "metadata": {},
   "source": [
    "```python\n",
    "# 假设一个客服机器人场景\n",
    "\n",
    "# 原始模型（过参数化）\n",
    "- 通用语言理解能力\n",
    "- 基础对话能力\n",
    "- 大量冗余参数\n",
    "\n",
    "# 特定任务适配（如手机客服）\n",
    "- 只需要学习：\n",
    "  - 手机相关专业词汇\n",
    "  - 特定问答模式\n",
    "  - 服务流程\n",
    "  ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cd457f4",
   "metadata": {},
   "source": [
    "&emsp;&emsp;通俗来讲LoRA的策略是：通过使用较小规模的矩阵来近似模拟大模型的原始矩阵。实现的方式是基于低秩分解的数学原理，通过较少的参数更新实现对大模型复杂功能的有效捕捉和适配，在减少计算资源消耗和提升微调效率的同时，保持或甚至提升模型对特定任务的适应性和性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9e58ef",
   "metadata": {},
   "source": [
    "$h = W_0x + \\Delta Wx = W_0x + ABx$<br>\n",
    "如果我是 $ d $ * $ d $的矩阵 就是 $W_0x=$ d  * $d $ <br>\n",
    "$\\Delta Wx=ABx$ <br>\n",
    "$A= $ d * $ r $ <br>\n",
    "$B= $ r * $ d $ <br>\n",
    "$h = W_0x + \\Delta Wx = W_0x + ABx =W_0x +  $ d * $ r  *  $ r * $ d  $ <br>\n",
    "&emsp;&emsp;我们可以做一下简单计算来直观感受Lora方法节省的内存：设ΔW有20，000行、30，000列，则它有6亿个参数。我们选择超参数r为8，那么实际所需的参数量为20，000x8 + 8x30，000 = 400，000\n",
    "，这是全参数量ΔW的1/1500.由此可见LORA的优化效果是非常可观的。这也让单卡进行微调训练成为可能。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3db5ea5b",
   "metadata": {},
   "source": [
    "## 2.2 Lora怎么配置"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344e6bb0",
   "metadata": {},
   "source": [
    "<div align=center><img src=\"https://typora-photo1220.oss-cn-beijing.aliyuncs.com/DataAnalysis/muyan/image-20241117171209788.png\" width=100%></div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "400e33db",
   "metadata": {},
   "source": [
    "初始化时候：<br>\n",
    "A 的元素除服从正态分布 N(0, 1), 例如:\n",
    "$$\n",
    "A=\n",
    "\\begin{pmatrix}\n",
    "0.5 & -0.2 & 0.1 & 0.3 & \\cdots & 0.7 \\\\\n",
    "-0.4 & 0.3 & -0.2 & 0.5 & \\cdots & -0.6 \\\\\n",
    "0.6 & 0.4 & -0.3 & -0.1 & \\cdots & 0.2 \\\\\n",
    "-0.5 & 0.4 & -0.3 & -0.1 & \\cdots & 0.5\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "B 的形式如下所示:\n",
    "$$\n",
    "B =\n",
    "\\begin{pmatrix}\n",
    "0 & 0 & 0 & 0 \\\\\n",
    "0 & 0 & 0 & 0 \\\\\n",
    "0 & 0 & 0 & 0 \\\\\n",
    "\\vdots & \\vdots & \\vdots & \\vdots \\\\\n",
    "0 & 0 & 0 & 0\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "训练<br>\n",
    "在训练过程中，仅更新A和B的值。\n",
    "随着训练的进⾏，B的值逐渐变得⾮零."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9798978a",
   "metadata": {},
   "source": [
    "1. 初始状态：\n",
    "- 矩阵B被初始化为0\n",
    "- 矩阵A被随机初始化\n",
    "- 此时权重更新ΔW (即B×A) 也为0\n",
    "\n",
    "2. 训练过程中B的变化原理：\n",
    "- 当模型接收到输入时，由于B初始为0，最初的权重更新没有效果\n",
    "- 但是通过反向传播，模型会计算损失对B的梯度\n",
    "- 这个梯度告诉我们如何调整B的值来减小损失\n",
    "- 通过梯度下降，B的值会逐步更新，偏离0\n",
    "- 随着训练进行，B会逐渐收敛到有助于完成任务的值"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84939306",
   "metadata": {},
   "source": [
    "3. 为什么B会变得非零：\n",
    "- 如果B保持为0，模型就无法学习任何任务特定的适配\n",
    "- 损失函数会推动B向着能够减小任务损失的方向变化\n",
    "- 这种变化是通过梯度下降自然发生的\n",
    "\n",
    "4. 影响B变化的因素：\n",
    "- 学习率：决定每次更新的步长\n",
    "- 任务性质：不同任务需要不同程度的适配\n",
    "- 优化器选择：不同优化器会影响B的收敛路径\n",
    "- 初始化方式：A的初始值会影响B的学习"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10538bb2",
   "metadata": {},
   "source": [
    "5. B的收敛过程：\n",
    "- 训练初期：B快速从0变化\n",
    "- 训练中期：变化速度放缓，开始形成有意义的特征\n",
    "- 训练后期：B的值趋于稳定"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26cf133c",
   "metadata": {},
   "source": [
    "问: 为什么初始化参数使用正态分布？<br>\n",
    "答：<br>\n",
    "&emsp;&emsp;确保初始梯度的有效传播：正态分布初始化有助于在训练初期确保梯度有效传播，避免梯度消失或爆炸的问题。 <br>\n",
    "&emsp;&emsp;提供足够的随机性：正态分布的随机初始化为模型提供了足够的随机性，从而能够探索更广泛的参数空间，增加了模型找到最优解的可能性。 <br>\n",
    "&emsp;&emsp;平衡训练初期的影响：正态初始化的值一般较小，结合 B 初始化为零矩阵，可以在训练初期确保新增的偏置矩阵对原始预训练权重的影响为零，从而避免破坏预训练模型的初始性能。<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "231c898a",
   "metadata": {},
   "source": [
    "## 2.3 Lora作用位置"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4857a5fd",
   "metadata": {},
   "source": [
    "1. 基本概念：\n",
    "- Q (Query): 查询向量，表示\"我想要查什么\"\n",
    "- K (Key): 键向量，表示\"我有什么信息\"\n",
    "- V (Value): 值向量，表示\"实际的信息内容\"\n",
    "- O (Output): 输出投影，将注意力的输出映射到所需的维度"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29bc4bd5",
   "metadata": {},
   "source": [
    "\n",
    "2. 在Transformer中的位置：\n",
    "```\n",
    "输入序列 \n",
    "   ↓\n",
    "Linear投影层\n",
    "   ↓\n",
    "Q、K、V矩阵变换\n",
    "   ↓\n",
    "注意力计算\n",
    "   ↓\n",
    "O矩阵变换（输出投影）\n",
    "   ↓\n",
    "输出结果\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c85993dd",
   "metadata": {},
   "source": [
    "3. 具体作用：\n",
    "- Q矩阵：将输入转换为查询形式\n",
    "- K矩阵：将输入转换为可被查询的键\n",
    "- V矩阵：存储实际的信息内容\n",
    "- O矩阵：将注意力机制的输出转换为所需的表示\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86259c9a",
   "metadata": {},
   "source": [
    "4. 在LoRA中的应用：\n",
    "- LoRA主要应用在这些权重矩阵的更新上\n",
    "- 对每个权重矩阵(Wq,Wk,Wv,Wo)都可以应用LoRA\n",
    "```python\n",
    "# 以Q为例\n",
    "原始: Q = input × Wq\n",
    "LoRA: Q = input × (Wq + BA)  # B和A是低秩矩阵\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7096860",
   "metadata": {},
   "source": [
    "\n",
    "5. LoRA的选择性应用：\n",
    "- 可以只对部分矩阵应用LoRA\n",
    "- 常见组合：\n",
    "  - 仅Q和V\n",
    "  - Q、K、V全部\n",
    "  - Q、K、V、O全部\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d53b2f6b",
   "metadata": {},
   "source": [
    "LORA 的作用在于$\\Delta W$<br>\n",
    "$ W_0 Q + \\Delta W q$<br>\n",
    "$ W_0 K + \\Delta W k$<br>\n",
    "$ W_0 V + \\Delta W v$<br>\n",
    "$ W_0 O + \\Delta W o$<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "181bcf66",
   "metadata": {},
   "source": [
    "\n",
    "6. 实际影响：\n",
    "- Q的更新影响查询方式\n",
    "- K的更新影响键的表示\n",
    "- V的更新影响值的内容\n",
    "- O的更新影响最终输出\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f145e0d",
   "metadata": {},
   "source": [
    "<div align=center><img src=\"https://typora-photo1220.oss-cn-beijing.aliyuncs.com/DataAnalysis/muyan/image-20241117181759857.png\" width=90%></div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3ee5fa5",
   "metadata": {},
   "source": [
    "GPT3使用LORA在不同类型注意力得到，分别在两份数据集上得到的效果，可以看到不限制参数预算情况下同时作用在$ W q、W k、W v、W o$ 效果最好。但是有限预算参数状况下$W q、W k $ 效果最好。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56da9334",
   "metadata": {},
   "source": [
    "<div align=center><img src=\"https://typora-photo1220.oss-cn-beijing.aliyuncs.com/DataAnalysis/muyan/image-20241117183212261.png\" width=100%></div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d734430",
   "metadata": {},
   "source": [
    "相同参数下训练$W q、W k $ 比单独训练$W q$效果要好，如果$W q$ 的效果想要更好就需要更大的参数量来微调。<br>\n",
    "同时可以看到$r$(Ranker)并非越大越好。建议值8-64,通常默认为8。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f17a4b5a",
   "metadata": {},
   "source": [
    "全参微调资源占用<br>\n",
    "1. 模型权重参数<br>\n",
    "参数数量：1B（10^9）每个参数占用4字节<br>\n",
    "总内存：10^9×4 字节 = 4GB<br>\n",
    "2. 优化器状态向量<br>\n",
    "AdamW优化器需要存储两个状态向量（一阶矩估计m和二阶矩估计v）每个状态向量占用4字节<br>\n",
    "总内存：10^9×4字节×2 = 8GB<br>\n",
    "3. 梯度<br>\n",
    "每个参数对应一个梯度每个梯度占用4字节<br>\n",
    "总内存：10^9 ×4 字节 = 4GB<br>\n",
    "总占用：4GB+8GB+4GB=16GB,可以粗略计算为4*NGB<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "037fb118",
   "metadata": {},
   "source": [
    "LoRA参数: 假设LoRA的rank设置为8，LoRA通过引入两个低秩矩阵（A和B）来进行微调。虽然具体的参数量依赖于模型的架构，但通常LoRA的可训练参数量远低于全参数微调。例如，LoRA可能只需要大约0.1B到0.2B的参数。\n",
    "\n",
    "LORA资源占用<br>\n",
    "假设LoRA的rank设置为8，LoRA通过引入两个低秩矩阵（A和B）来进行微调。虽然具体的参数量依赖于模型的架构，但通常LoRA的可训练参数量远低于全参数微调。例如，LoRA可能只需要大约0.1B到0.2B的参数。\n",
    "1. 模型权重参数<br>\n",
    "参数数量：1B（10^9）每个参数占用4字节<br>\n",
    "总内存：10^9×4 字节 = 4GB<br>\n",
    "2. 优化器状态向量<br>\n",
    "AdamW优化器需要存储两个状态向量（一阶矩估计m和二阶矩估计v）每个状态向量占用4字节<br>\n",
    "总内存：16M×4 bytes×2=128 MB<br>\n",
    "3. 梯度<br>\n",
    "每个参数对应一个梯度每个梯度占用4字节<br>\n",
    "总内存：16M×4 bytes×1=64MB<br>\n",
    "总占用：4GB+64MB+128 MB=4.2GB<br>\n",
    "\n",
    "这个也仅限于参数估量计算，不能完全作为生产使用，生产环境有很多影响资源大小的因素如批处理大小、优化器状态等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c788699d",
   "metadata": {},
   "source": [
    "# 3.Lora 代码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fdc77fd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0, Loss: 1.099842\n",
      "Iteration 100, Loss: 0.589540\n",
      "Iteration 200, Loss: 0.439351\n",
      "Iteration 300, Loss: 0.435789\n",
      "Iteration 400, Loss: 0.435765\n",
      "Iteration 500, Loss: 0.435765\n",
      "Iteration 600, Loss: 0.435765\n",
      "Iteration 700, Loss: 0.435765\n",
      "Iteration 800, Loss: 0.435765\n",
      "Iteration 900, Loss: 0.435765\n",
      "\n",
      "原始参数量: 100\n",
      "LoRA参数量: 40\n",
      "参数减少: 60.00%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABcAAAAHqCAYAAAAjy+2hAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAACPeklEQVR4nOzdd3wUdf7H8fem9wAJCSABAlFAEKUoAtLkjIJiR2woCp4cWBArhwU5NYcFsRxwKIgNRU+x/E7RnEgTVECwoSg1ARJCTYW0nd8fmNU1IWTDJDPZeT3vMY9zJ7Oz790s+WQ/+c736zIMwxAAAAAAAAAAAH4mwOoAAAAAAAAAAADUBRrgAAAAAAAAAAC/RAMcAAAAAAAAAOCXaIADAAAAAAAAAPwSDXAAAAAAAAAAgF+iAQ4AAAAAAAAA8Es0wAEAAAAAAAAAfokGOAAAAAAAAADAL9EABwAAAAAAAAD4JRrggM25XK4abUuWLDmux5k8ebJcLlet7rtkyRJTMhzPY//nP/+p98cGAAAAAACAvQVZHQBA9VatWuV1+x//+Ic+//xzLV682Gv/ySeffFyPM3r0aJ133nm1um+3bt20atWq484AAAAAAAAAmIkGOGBzZ555ptftpk2bKiAgoNL+PysqKlJERESNH6dly5Zq2bJlrTLGxMQcMw8AAAAAAABQ35gCBfADAwYMUOfOnbVs2TL17t1bERERuvHGGyVJCxYsUGpqqpo3b67w8HB17NhR9913nwoLC73OUdUUKG3atNEFF1ygRYsWqVu3bgoPD1eHDh00d+5cr+OqmgJl5MiRioqK0qZNmzRkyBBFRUUpKSlJd955p4qLi73uv2PHDl1++eWKjo5Wo0aNdM0112j16tVyuVyaN2+eKa/RDz/8oIsuukiNGzdWWFiYTjvtNL388stex7jdbj3yyCNq3769wsPD1ahRI3Xp0kXPPPOM55g9e/bor3/9q5KSkhQaGqqmTZuqT58++t///mdKTgAAAAAAAJiHEeCAn8jKytK1116re+65R4899pgCAo78fevXX3/VkCFDNH78eEVGRurnn3/W1KlT9fXXX1eaRqUq3377re68807dd999SkxM1IsvvqhRo0YpJSVF/fr1q/a+paWluvDCCzVq1CjdeeedWrZsmf7xj38oNjZWDz74oCSpsLBQAwcO1P79+zV16lSlpKRo0aJFGj58+PG/KL/ZuHGjevfurYSEBD377LOKi4vTa6+9ppEjR2r37t265557JEmPP/64Jk+erPvvv1/9+vVTaWmpfv75Zx08eNBzrhEjRuibb77Ro48+qpNOOkkHDx7UN998o3379pmWFwAAAAAAAOagAQ74if379+vtt9/W2Wef7bX//vvv9/y3YRjq06ePOnbsqP79++u7775Tly5dqj3v3r179cUXX6hVq1aSpH79+umzzz7T/Pnzj9kALykp0cMPP6xhw4ZJkgYNGqQ1a9Zo/vz5ngb4yy+/rE2bNunjjz/2zEGempqqoqIi/fvf//btRTiKyZMnq6SkRJ9//rmSkpIkSUOGDNHBgwf18MMP6+abb1ZsbKy++OILnXLKKZo8ebLnvueee67Xub744guNHj1aN910k2ffRRddZEpOAAAAAAAAmIspUAA/0bhx40rNb0nasmWLrr76ajVr1kyBgYEKDg5W//79JUk//fTTMc972mmneZrfkhQWFqaTTjpJ27dvP+Z9XS6Xhg4d6rWvS5cuXvddunSpoqOjKy3AedVVVx3z/DW1ePFiDRo0yNP8rjBy5EgVFRV5Fho944wz9O2332rs2LH65JNPlJeXV+lcZ5xxhubNm6dHHnlEX375pUpLS03LCQAAAAAAAHPRAAf8RPPmzSvtKygoUN++ffXVV1/pkUce0ZIlS7R69Wq9++67kqRDhw4d87xxcXGV9oWGhtbovhEREQoLC6t038OHD3tu79u3T4mJiZXuW9W+2tq3b1+Vr0+LFi08X5ekiRMn6sknn9SXX36pwYMHKy4uzjNqvcKCBQt0/fXX68UXX1SvXr3UpEkTXXfddcrOzjYtLwAAAAAAAMxBAxzwE39ewFI6MvJ5165dmjt3rkaPHq1+/fqpR48eio6OtiBh1eLi4rR79+5K+81sKMfFxSkrK6vS/l27dkmS4uPjJUlBQUGaMGGCvvnmG+3fv19vvPGGMjMzde6556qoqMhz7PTp07Vt2zZt375daWlpevfddzVy5EjT8gIAAAAAAMAcNMABP1bRFA8NDfXab9bc2mbo37+/8vPz9fHHH3vtf/PNN017jEGDBnn+GPBHr7zyiiIiInTmmWdWuk+jRo10+eWXa9y4cdq/f7+2bdtW6ZhWrVrplltu0TnnnKNvvvnGtLwAAAAAAAAwB4tgAn6sd+/eaty4scaMGaOHHnpIwcHBev311/Xtt99aHc3j+uuv19NPP61rr71WjzzyiFJSUvTxxx/rk08+kSQFBNTs73Rffvlllfv79++vhx56SP/3f/+ngQMH6sEHH1STJk30+uuv67///a8ef/xxxcbGSpKGDh2qzp07q0ePHmratKm2b9+u6dOnq3Xr1jrxxBOVm5urgQMH6uqrr1aHDh0UHR2t1atXa9GiRbr00kvNeUEAAAAAAABgGhrggB+Li4vTf//7X91555269tprFRkZqYsuukgLFixQt27drI4nSYqMjNTixYs1fvx43XPPPXK5XEpNTdWMGTM0ZMgQNWrUqEbneeqpp6rc//nnn2vAgAFauXKl/v73v2vcuHE6dOiQOnbsqJdeeslr6pKBAwfqnXfe0Ysvvqi8vDw1a9ZM55xzjh544AEFBwcrLCxMPXv21Kuvvqpt27aptLRUrVq10r333qt77rnHhFcDAAAAAAAAZnIZhmFYHQIA/uyxxx7T/fffr4yMDLVs2dLqOAAAAAAAAGiAGAEOwHLPP/+8JKlDhw4qLS3V4sWL9eyzz+raa6+l+Q0AAAAAAIBaowEOwHIRERF6+umntW3bNhUXF3umFbn//vutjgYAAAAAAIAGjClQAAAAAAAAAAB+KcDqAAAAAAAAAAAA1AUa4AAAAAAAAAAAv0QDHAAAAAAAAADglxy3CKbb7dauXbsUHR0tl8tldRwAAOqdYRjKz89XixYtFBDA38IBAPXn8OHDKikpMfWcISEhCgsLM/WcAAD4M6fVY8c1wHft2qWkpCSrYwAAYLnMzEy1bNnS6hgAAIc4fPiwkltHKTun3NTzNmvWTFu3brXth24AAOzEifXYcQ3w6OhoSUc+9MfExFicBgCA+peXl6ekpCRPTQQAoD6UlJQoO6dc29e2UUy0OVcg5eW71br7NpWUlNjyAzcAAHbjxHrsuAZ4xbQnMTExNMABAI7GVGAAACtERbsUFW1ODXKLWgYAQG04qR4z8ScAAIAf+/LLLzVs2DA1b95cISEhatasmS6//HKtWrXKp/NMnjy51n80WbJkiVwul5YsWVKr+9fUgAEDNGDAgGqPueCCCxQdHa2ysjKv/evWrZPL5VLz5s0r3Wf58uVyuVx69tlnfcrjcrk0efJkn+5ToU2bNrrggguOedyGDRs0efJkbdu2rUbnnTdvnlwu11G/H4ZhKCUlRS6X65iv5dHMmDFD8+bN8+k+9fUeAQA7qvjZvGbNmuM+V8XP04otMDBQTZs21dChQ6s9/wcffCCXy6W4uDgVFxfX6rEvvfRSuVwu3XLLLbWNb2vHU9eP19Fq67Zt2+RyuXyuu4DT0AAHAADwU88995z69OmjHTt26PHHH9f//vc/Pfnkk9q5c6fOOussPf/88zU+1+jRo31umlfo1q2bVq1apW7dutXq/mYaOHCgCgoKKjUBlixZosjISGVnZ+vnn3+u9LWK+/pi1apVGj169HHlPZYNGzbo4YcfrnEDvEJ0dLTmzJlTaf/SpUu1efPm45oiqTYNcDu9R1D3yg23qRuAyh577DGtWrVKS5Ys0QMPPKCVK1eqf//++vXXX6s8vqIm7N+/X++9957Pj5eTk6P/+7//kyS9/vrrOnz4cK2z21V91PWjOVptbd68uVatWqXzzz+//kOhwXNSPaYBDgAA4Ie++OILjR8/XkOGDNHy5cs1YsQI9evXT9dee62WL1+uIUOG6Pbbb9cXX3xR7XmKiookSS1bttSZZ55ZqywxMTE688wzbTH9XEUT+88jjZcsWaKLLrpIzZs31+eff17pa/Hx8ercubNPj3XmmWfadqHZ4cOH65133lFeXp7X/jlz5qhXr15q1apVveQoLS1VWVmZrd4jqHtuGaZuACo78cQTdeaZZ6pv37667bbb9PTTT6uoqEivvfZapWOzs7P10Ucf6eyzz1ZYWFiVfyA9lldeeUWlpaU6//zzdfDgQb377rtmPI1aKS8vr/Uo9urYsa6HhobqzDPPVNOmTa2OggbISfWYBjgAAIAfSktLk8vl0syZMxUU5L3sS1BQkGbMmCGXy6V//vOfnv0V05x88803uvzyy9W4cWO1a9fO62t/VFxcrDvvvFPNmjVTRESE+vXrp7Vr16pNmzYaOXKk57iqprcYOXKkoqKitGnTJg0ZMkRRUVFKSkrSnXfeWelD68MPP6yePXuqSZMmiomJUbdu3TRnzhwZhu+/aJ922mlq3LixVxa3263ly5drwIAB6t+/v1cDvKSkRKtWrdKAAQM8zz87O1s333yzWrZsqZCQECUnJ+vhhx+uNK1KVZdKr1ixQr169VJYWJhOOOEEPfDAA3rxxRflcrmqHMW9aNEidevWTeHh4erQoYPmzp3r+dq8efM0bNgwSUca+xWXu9dk9PVVV10lSXrjjTc8+3Jzc/XOO+/oxhtvrPI+Nfk+tGnTRj/++KOWLl3qydOmTRtJv78PXn31Vd1555064YQTFBoaqk2bNlV6j+zdu1dJSUnq3bu3SktLPeffsGGDIiMjNWLEiGM+RwDwNytWrNCgQYMUHR2tiIgI9e7dW//9739rdN8ePXpIknbv3l3pay+//LLKysp0xx136NJLL9Vnn32m7du3+5Rt7ty5SkxM1Msvv6zw8HCvelWhYqqX9PR03XDDDWrSpIkiIyM1dOhQbdmyxevYAQMGqHPnzlq+fLnOPPNMhYeHe+pmeXm557iKKUAef/xxPfLII0pOTlZoaKinln/wwQfq1auXIiIiFB0drXPOOcfrirY333xTLper0lVxDz30kAIDA5Wenu7Z9+e6XvF8Fi9erJtuuklxcXGKiYnRddddp8LCQmVnZ+uKK65Qo0aN1Lx5c911111eNU06/tp6tClQavJeqcj/+eef629/+5vi4+MVFxenSy+9VLt27ar0/QMaMhrgAAAAfqa8vFyff/65evTocdSRSklJSerevbsWL17s9UFSOjKHZ0pKit5++23NmjXrqI9zww03aPr06brhhhv0/vvv67LLLtMll1yigwcP1ihnaWmpLrzwQg0aNEjvv/++brzxRj399NOaOnWq13Hbtm3TzTffrLfeekvvvvuuLr30Ut166636xz/+UaPH+aOAgAD169dPK1as8DSs169frwMHDqh///7q37+/li5d6jn+yy+/1KFDhzwjx7Ozs3XGGWfok08+0YMPPqiPP/5Yo0aNUlpamm666aZqH/u7777TOeeco6KiIr388suaNWuWvvnmGz366KNVHv/tt9/qzjvv1B133KH3339fXbp00ahRo7Rs2TJJ0vnnn6/HHntMkvSvf/1Lq1atqvFl0DExMbr88su9GhRvvPGGAgICNHz48CrvU5Pvw8KFC9W2bVt17drVk2fhwoVe55k4caIyMjI0a9Ysffjhh0pISKj0WPHx8XrzzTe1evVq3XvvvZKOXI0wbNgwtWrVqtr3JezPbfL/ACdYunSpzj77bOXm5mrOnDl64403FB0draFDh2rBggXHvP/WrVslSSeddFKlr82dO1fNmzfX4MGDdeONN8rtdvs0ldXKlSv1008/6brrrlNcXJwuu+wyLV682POYfzZq1CgFBARo/vz5mj59ur7++msNGDCg0u8P2dnZuvLKK3XNNdfo/fff1+WXX65HHnlEt99+e6VzPvvss1q8eLGefPJJffzxx+rQoYPmz5+viy66SDExMXrjjTc0Z84cHThwQAMGDNCKFSskSVdeeaXGjBmjO++80zM92uLFi/XII4/o73//u84555xjPv/Ro0crNjZWb775pu6//37Nnz9fN910k84//3ydeuqp+s9//qPrr79eTz31lJ577jmv+5pVW//I1/fK6NGjFRwcrPnz5+vxxx/XkiVLdO211x7zeaPhc1Q9NhwmNzfXkGTk5uZaHQUAAEtQC/1fdna2Icm48sorqz1u+PDhhiRj9+7dhmEYxkMPPWRIMh588MFKx1Z8rcKPP/5oSDLuvfder+PeeOMNQ5Jx/fXXe/Z9/vnnhiTj888/9+y7/vrrDUnGW2+95XX/IUOGGO3btz9q5vLycqO0tNSYMmWKERcXZ7jdbs/X+vfvb/Tv37/a52wYhjF9+nRDkrFy5UrDMAzjqaeeMpo3b24YhmFs2LDBkGT88MMPhmEYxsMPP2xIMjZs2GAYhmHcfPPNRlRUlLF9+3avcz755JOGJOPHH3/07JNkPPTQQ57bw4YNMyIjI409e/Z4PZ+TTz7ZkGRs3brVs79169ZGWFiY1+McOnTIaNKkiXHzzTd79r399tuVXtvqvPTSS4YkY/Xq1Z7vS8VzPf30042RI0cahmEYnTp1qva1rO77cLT7Vjxev379jvq1Pz+PqVOnGpKMhQsXGtdff70RHh5ufPfddzV6rrCfivqT+fMJRu7OJFO2zJ9PoKahwfvjz+ajOfPMM42EhAQjPz/fs6+srMzo3Lmz0bJlS8/P4YqfpwsWLDBKS0uNoqIi44svvjDat29vnHzyycaBAwe8zrts2TJDknHfffcZhmEYbrfbSE5ONlq3bu31s706N954oyHJ+Omnn7wyPPDAA1U+z0suucRr/xdffGFIMh555BHPvv79+xuSjPfff9/r2JtuuskICAjw1MetW7cakox27doZJSUlnuPKy8uNFi1aGKeccopRXl7u2Z+fn28kJCQYvXv39uw7fPiw0bVrVyM5OdnYsGGDkZiYaPTv398oKyvzeuw/1/WK53Prrbd6HXfxxRcbkoxp06Z57T/ttNOMbt26Vf0iGrWrrRXP/6WXXvLsq+l7pSL/2LFjvc75+OOPG5KMrKyso2ZFw+bEeswIcAAAAIcyfru89s9Tm1x22WXHvG/FKOkrrrjCa//ll19eacqVo3G5XBo6dKjXvi5dulS67Hrx4sX6y1/+otjYWAUGBio4OFgPPvig9u3bp5ycnBo91h/9eR7wJUuWqH///pKkjh07KiEhwXPp9JIlS5SYmKiOHTtKkv7v//5PAwcOVIsWLVRWVubZBg8eLEleo8f/rGJEVnx8vGdfQEBApdewwmmnneY1F3dYWJhOOukkny9LP5r+/furXbt2mjt3rr7//nutXr36qNOfSOZ9H2ry/qpw99136/zzz9dVV12ll19+Wc8995xOOeWUGt8fAPxBYWGhvvrqK11++eWKiory7A8MDNSIESO0Y8cObdy40es+w4cPV3BwsCIiItSnTx/l5eXpv//9rxo1auR1XMV83xU//10ul0aOHKnt27frs88+O2a2goICvfXWW+rdu7c6dOgg6ff6Mm/ePLndlUeFXnPNNV63e/furdatW1dagyM6OloXXnih176rr75abrfbczVUhQsvvFDBwcGe2xs3btSuXbs0YsQIBQT83vqKiorSZZddpi+//NKzzkloaKjeeust7du3T926dZNhGHrjjTcUGBh4zOcvSRdccIHX7YrfGf58RVbHjh3r/Hec2rxX/vwad+nSRZJM+30DsAMa4AAAAH4mPj5eERERR730uMK2bdsUERGhJk2aeO1v3rz5MR9j3759kqTExESv/UFBQYqLi6tRzoiICIWFhXntCw0N1eHDhz23v/76a6WmpkqSXnjhBX3xxRdavXq1Jk2aJEk6dOhQjR7rj0455RTFx8fr888/98z/XdEAl6R+/fppyZIlKi4u1qpVqzwNc+nI3KkffvihgoODvbZOnTpJOjJ39dHs27ev0uslVX4NK1T1OoaGhtbqOVfF5XLphhtu0GuvvaZZs2bppJNOUt++fas81szvQ03eX3/MOHLkSB0+fFjNmjVj7m8/4aRFtwAzHDhwQIZhVPnzs0WLFpJ+r8sVpk6dqtWrV2vp0qWaNGmSdu/erYsvvthrnY38/Hy9/fbbOuOMM9S0aVMdPHhQBw8e1CWXXCKXy1WjxTAXLFiggoICXXHFFZ775+bm6oorrlBmZqbXHNoVmjVrVuW+Pz+HqupjxX3/fOyfX5uKrx/tNXO73Tpw4IBnX0pKivr27avDhw/rmmuu8alW/fn3qJCQkKPur+vfcWrzXvnz7xuhoaG1fnw0LE6qxzUbngMAAIAGIzAwUAMHDtSiRYu0Y8eOKucB37Fjh9auXavBgwdXGuH05xHhVan4sLR7926dcMIJnv1lZWWVPlgdjzfffFPBwcH6v//7P69m+XvvvVfrc7pcLvXv31+LFi3S119/rYMHD3o1wPv376/Jkydr1apVOnz4sFcDPD4+Xl26dDnqvN0VHy6rEhcXV+XiY9nZ2bV+Lsdr5MiRevDBBzVr1qyjPifJ3O9DTd5fFbKysjRu3Diddtpp+vHHH3XXXXfp2Wef9fkxYS9uGSo36YOy3T9wA2Zo3LixAgIClJWVVelrFYsV/vHqIklq27atZ+HLfv36KTw8XPfff7+ee+453XXXXZKOrP1QVFSkr7/+Wo0bN6507oULF+rAgQNVfq1CRZN8/PjxGj9+fJVfP/fcc732VVX3srOzlZKS4rWvupr556btn2tLxdeP9poFBAR4Pa8XX3xR//3vf3XGGWfo+eef1/Dhw9WzZ89K9zVTXfyOU5v3CpzLSfWYEeAAAAB+aOLEiTIMQ2PHjq20yGV5ebn+9re/yTAMTZw4sVbn79evnyRVWkzpP//5j2dxSTO4XC4FBQV5NekPHTqkV1999bjOO3DgQBUWFuqJJ55QQkKC53Jl6UgDfN++fZ6Fqv7YAL/gggv0ww8/qF27durRo0elrboGeP/+/bV48WKvUeJut1tvv/12rZ/H8Y7SOuGEE3T33Xdr6NChuv766496nC/fB7NGqZeXl+uqq66Sy+XSxx9/rLS0ND333HN69913j/vcANCQREZGqmfPnnr33Xe9fr663W699tpratmyZZWLW/7RPffco5SUFP3zn/9Ufn6+pCPN6ejoaH322Wf6/PPPvbYnnnhCxcXFev311496zp9++kmrVq3SZZddVun+n3/+uWeR6z//YfzP51y5cqW2b9+uAQMGeO3Pz8/XBx984LVv/vz5ngWtq9O+fXudcMIJmj9/vmfKN+nIFCHvvPOOevXqpYiICEnS999/r9tuu03XXXedli9fri5dumj48OFeI8TrQl3UVjPeK4A/YgQ4AACAH+rTp4+mT5+u8ePH66yzztItt9yiVq1aKSMjQ//617/01Vdfafr06erdu3etzt+pUyddddVVeuqppxQYGKizzz5bP/74o5566inFxsZ6zbd5PM4//3xNmzZNV199tf76179q3759evLJJz2N39qqaGovXLhQl19+udfXOnfurLi4OC1cuFAnnHCCTjzxRM/XpkyZovT0dPXu3Vu33Xab2rdvr8OHD2vbtm366KOPNGvWrCpH3EvSpEmT9OGHH2rQoEGaNGmSwsPDNWvWLBUWFkpSrV6zzp07S5Jmz56t6OhohYWFKTk5ucbT0EjSP//5z2Me48v34ZRTTtGbb76pBQsWqG3btgoLC6vVvN0PPfSQli9frk8//VTNmjXTnXfeqaVLl2rUqFHq2rWrkpOTfT4n7MHMS6XtPuIM8MXixYu1bdu2SvuHDBmitLQ0nXPOORo4cKDuuusuhYSEaMaMGfrhhx/0xhtvHPPqmuDgYD322GO64oor9Mwzz+jiiy/W119/rb/97W86++yzKx3fp08fPfXUU5ozZ45uueWWKs9ZMfr7nnvu0RlnnFHp6/n5+frss8/02muv6fbbb/fsX7NmjUaPHq1hw4YpMzNTkyZN0gknnKCxY8d63T8uLk5/+9vflJGRoZNOOkkfffSRXnjhBf3tb3/zWiOjKgEBAXr88cd1zTXX6IILLtDNN9+s4uJiPfHEEzp48KCn9hUWFuqKK65QcnKyZsyYoZCQEL311lvq1q2bbrjhhuMajX0sdVVbj/e9AudwUj22dAT4smXLNHToULVo0UIul+uYP1iysrJ09dVXq3379goICKjy8hoAAAAcceutt+qLL75Qy5Ytdeedd+rss8/WhAkT1Lx5c61YsUK33nrrcZ3/pZde0u233645c+Zo6NChevPNN/XWW29JUqVFtmrr7LPP9izSOHToUE2aNEmXX3657rvvvuM678knn6xmzZrJMAyv6U+kIyOy+vbtK8MwKo1Ga968udasWaPU1FQ98cQTOu+88zRixAjNnTtXp512WrWXiZ966qlKT09XeHi4rrvuOv31r39Vp06dPB/4Y2NjfX4eycnJmj59ur799lsNGDBAp59+uj788EOfz3MsvnwfHn74YfXv31833XSTzjjjjEoLndZEenq60tLS9MADD2jQoEGe/fPmzVNMTIyGDx+ukpKS43pOAGA39957r4YNG1Zpy8nJ8VxFFBkZqZEjR+rKK69Ubm6uPvjgAw0fPrxG5x82bJh69uypadOmeZrXN998c5XHBgcHa+TIkVq/fr2++eabSl8vLS3Vq6++qtNOO63K5rd0pHHfsmXLSnOJz5kzRyUlJbryyit12223qUePHlqyZEmlObObNWum+fPn6+WXX9aFF16ot956S3//+99rPBXW1Vdfrffee0/79u3T8OHDdcMNNygmJkaff/65zjrrLEnSmDFjlJGRobfffluRkZGSjkwf8+KLL+r999/X9OnTa/RYtVFXtdWM9wrgb1zGH68FqWcff/yxvvjiC3Xr1k2XXXaZFi5cqIsvvviox2/btk1PP/20unfvrqefflr9+/f3+YdRXl6eYmNjlZubq5iYmON7AgAANEDUQtSllStXqk+fPnr99dd19dVXWx2nQUhNTdW2bdv0yy+/WB0FqFMV9eeXnxIVHW3OWKz8fLdO6ribmgY0APPmzdMNN9yg1atXe+YnP5oBAwZo7969+uGHH+opHeAcTqzHlk6BMnjwYA0ePLjGx7dp00bPPPOMJGnu3Ll1FQsAAAA1kJ6erlWrVql79+4KDw/Xt99+q3/+85868cQTdemll1odz5YmTJigrl27KikpSfv379frr7+u9PT0SqPjAH/m/m0z61wAAMB3TqrHzAF+nIrLyvXr7gLlHSpV7xRW0gUAAM4RExOjTz/9VNOnT1d+fr7i4+M1ePBgpaWlKSwszOp4tlReXq4HH3xQ2dnZcrlcOvnkk/Xqq6/q2muvtToaAAAA4Jf8vgFeXFys4uJiz+28vDxTz//9jlxdPmuVWsSGaeXEQce+AwAAgJ/o2bOnVqxYYXWMBuWZZ57xXNEIOFW5DJWbtFiWWecBUPdGjhypkSNH1ujYJUuW1GkWAM6qx5Yuglkf0tLSFBsb69mSkpJMPX9KQpQkaVfuYRUWl5l6bgAAAADwN+WGuRsAAPCdk+qx3zfAJ06cqNzcXM+WmZlp6vkbRYQoPipEkrR5T4Gp5wYAAAAAAAAA1J7fT4ESGhqq0NDQOn2Mdk2jtLdgvzblFKhLy0Z1+lgAAAAA0JA5adEtAADsykn12NIGeEFBgTZt2uS5vXXrVq1fv15NmjRRq1atNHHiRO3cuVOvvPKK55j169d77rtnzx6tX79eISEhOvnkk+s7vkdKQpS+2nqkAQ4AAI7N7XZr165dio6OlsvlsjoOAOAPDMNQfn6+WrRooYAAv79o2PGoyQBgT/5aj9PS0vTuu+/q559/Vnh4uHr37q2pU6eqffv2dfaYljbA16xZo4EDB3puT5gwQZJ0/fXXa968ecrKylJGRobXfbp27er577Vr12r+/Plq3bq1tm3bVi+Zq1IxD/ivNMABAKiRXbt2mb4uBwDAXJmZmWrZsqXp53XLpXKZ02h1m3QeJ6MmA4C9+Vs9Xrp0qcaNG6fTTz9dZWVlmjRpklJTU7VhwwZFRkaakufPLG2ADxgwQIZx9FnS582bV2lfdcdbpaIBvpkGOAAANRIdHS1JuvmTIQqJDLY4zRFrpnSzOoKXHYMCrY7gJXSvvUadlMTb60LL5svtlUeSimPt9R76/KE5VkfwclPGWVZH8PLN4g5WR/BwFx/WtqemeH5Wm35+48hm1rlwfCq+zxP+N0ihkfaYJXX+Z/2sjuClffftVkfwsnFta6sjeAk4odDqCF4aL4qwOkIljb89YHUELy8ufN3qCF76Lb3Z6ghejHJ7/HHVfeiwdt31T7+rx4sWLfK6/dJLLykhIUFr165Vv3518/PfHtWtgatogG/fX6SSMrdCguz1AREAALupuMQ6JDJYoVH2aIAHBYdZHcFLQLi9mpeBofb6/SYgzF4N56Bge+WRpLIQe72HYqLt9R4KjgyxOoKXwDB7/QySxHQYDlHxfQ6NDFKYTWpygM3+Pdjt54XdXp+AiHKrI3gJDLHX6yNJQYF1uzadr6JtVpMDwu31PbNLA7xCQ6rHeXl5XrdrsjZjbm6uJKlJkyZ1lste7/gGqllMmKJCg1TuNrRtn73+8gkAAAAAdlL+2yXXZm0AAMB3dVGPk5KSFBsb69nS0tKqzWAYhiZMmKCzzjpLnTt3rrPnyghwE7hcLrVrGqlvd+RqU06BTkqsm0sTAAAAAKChM7NxTQMcAIDaqYt6nJmZqZiYGM/+Y43+vuWWW/Tdd99pxYoVpuQ4GhrgJmmXEOVpgAMAAAAAAACAk8TExHg1wKtz66236oMPPtCyZcvqZJHPP6IBbpKKecBpgAMAAADA0bkNl9yGOSPOzDoPAABOY1U9NgxDt956qxYuXKglS5YoOTnZlAzVoQFukpSmNMABAAAA4FiYAgUAAOtZVY/HjRun+fPn6/3331d0dLSys7MlSbGxsQoPDzclz5+xCKZJKkaAb9lbILfbsDgNAAAAAAAAANjLzJkzlZubqwEDBqh58+aebcGCBXX2mIwAN0mrJhEKCQzQ4VK3dh48pKQmEVZHAgAAAADbKVeAyk0ai1VuylkAAHAeq+qxYdT/wGFGgJskKDBAbeKPNL2ZBgUAAAAAAAAArEcD3EQshAkAAAAA1TN+W3TLjM1gEUwAAGrFSfWYKVBMxEKYAAAAAFA9FsEEAMB6TqrHjAA3UUpitCRp0x4a4AAAAAAAAABgNRrgJvrjCHArJnQHAKC+zZgxQ8nJyQoLC1P37t21fPlyqyMBAGyu3AgwdQP1GADgOyfVY3una2DaNo2UyyXlHirV3oISq+MAAFCnFixYoPHjx2vSpElat26d+vbtq8GDBysjI8PqaAAAG3PLJbcCTNrsfcl1faAeAwBqw0n1mAa4icKCA5XUOEIS84ADAPzftGnTNGrUKI0ePVodO3bU9OnTlZSUpJkzZ1odDQAAx6AeAwBQPRrgJktJ+G0aFOYBBwD4sZKSEq1du1apqale+1NTU7Vy5cpKxxcXFysvL89rAwA4U8WiW2ZtTuZrPZaoyQCAI5xUj2mAm6yiAb6ZEeAAAD+2d+9elZeXKzEx0Wt/YmKisrOzKx2flpam2NhYz5aUlFRfUQEA8Fu+1mOJmgwAcB4a4Cb740KYAAD4O5fL+y/9hmFU2idJEydOVG5urmfLzMysr4gAAJtx0qJb9aWm9ViiJgMAjnBSPQ6yOoC/aZdAAxwA4P/i4+MVGBhYaXRZTk5OpVFokhQaGqrQ0ND6igcAsLEji26Zc6m03Rfdqmu+1mOJmgwAOMJJ9dje7fkGqGIKlOy8w8o/XGpxGgAA6kZISIi6d++u9PR0r/3p6enq3bu3RakAAHAW6jEAAMfGCHCTxYYHq2l0qPbkF2vznkKdltTI6kgAANSJCRMmaMSIEerRo4d69eql2bNnKyMjQ2PGjLE6GgDAxtwKULlJY7HcMkw5T0NGPQYA1IaT6jEN8DqQ0jRKe/KLtSmngAY4AMBvDR8+XPv27dOUKVOUlZWlzp0766OPPlLr1q2tjgYAsDEz5wotN+z9gbs+UI8BALXhpHpMA7wOpCREadWWfcwDDgDwe2PHjtXYsWOtjgEAgKNRjwEAODoa4HUghYUwAQAAAKBKbgXI7ZBLrgEAsCsn1WMWwawDFQ3wzXtogAMAAAAAAACAVRgBXgcqGuDb9xWquKxcoUGBFicCAAAAAHsoN1wqN1ymnQsAAPjOSfWYBngdSIgOVXRokPKLy7Rtb5HaN4u2OhIAAAAA2EK5AlRu0sXI5Ta/5BoAALtyUj1mCpQ64HK51I55wAEAAAAAAADAUowAryMpCVFan3mQBjgAAAAA/IHbCJDbMGnRLcPeI84AALArJ9VjGuB1pGIe8E0shAkAAAAAHk665BoAALtyUj1mCpQ6ktKUKVAAAAAAAAAAwEqMAK8jFSPAt+wpULnbUGCAvVdDBQAAAID64JZUbpjz+chtylkAAHAeJ9VjGuB1JKlJhEKCAlRc5taOA0VqHRdpdSQAAAAAsJxbAXKbdDGyWecBAMBpnFSP7Z2uAQsMcKlt/JGmN9OgAAAAAAAAAED9YwR4HWqXEKWfs/O1KadAgzomWh0HAADbWfnM6QoMDrM6hiQpIMrqBN46Pr3L6gheNj7WxOoIXtyH7PVrbIu7t1gdoZLvslpYHcFL18fGWh3BS1mE1Qm8lZxgn4uH3YfqNku5EaByw6RFt0w6D6T/TT1LQTapye4Li62O4GXHG8lWR/DSblim1RG8/PKLverNBfcssTpCJYt2nWx1BC/9p99ldQQvQU3ttYBi3Kk5VkeQJJUXFmtHXZ7fQfXY3ukaOBbCBAAAAAAAAADr2GvojJ+pWAhz0x4a4AAAAAAgSW655JZZi26Zcx4AAJzGSfWYBngd8jTAcwpkGIZcLnu/GQAAAACgrjnpkmsAAOzKSfXY3ukauOT4SAW4pPzDZdqTb685zAAAAAAAAADA3zECvA6FBQcqqUmEtu8r0qacAiXE2GNBEQAAAACwSrkCVG7SWCyzzgMAgNM4qR7bO50f8CyEyTzgAAAAACC34TJ1AwAAvnNSPaYBXsf+OA84AAAAAAAAAKD+MAVKHWtHAxwAAAAAPNwmXnLtZkwXAAC14qR6bO90fuBEGuAAAAAAAAAAYAlGgNexihHgOfnFyjtcqpiwYIsTAQAAAIB13EaA3IZJI85MOg8AAE7jpHpsabply5Zp6NChatGihVwul957771j3mfp0qXq3r27wsLC1LZtW82aNavugx6HmLBgJcaESmIUOAAAAACUy2XqBgAAfOekemxpA7ywsFCnnnqqnn/++Rodv3XrVg0ZMkR9+/bVunXr9Pe//1233Xab3nnnnTpOenxYCBMAAAAAAAAA6p+lU6AMHjxYgwcPrvHxs2bNUqtWrTR9+nRJUseOHbVmzRo9+eSTuuyyy+oo5fFLaRqlLzbt02Ya4AAAAAAczkmXXAMAYFdOqsf2Tvcnq1atUmpqqte+c889V2vWrFFpaWmV9ykuLlZeXp7XVt8YAQ4AAAAAR5TLzMuuay4tLU2nn366oqOjlZCQoIsvvlgbN26sq6cJAICtWVWPrdCgGuDZ2dlKTEz02peYmKiysjLt3bu3yvukpaUpNjbWsyUlJdVHVC8VC2Fu2kMDHAAAAACssHTpUo0bN05ffvml0tPTVVZWptTUVBUWFlodDQAA1CFLp0CpDZfLe1J1wzCq3F9h4sSJmjBhgud2Xl5evTfBK0aAZ+4v0uHScoUFB9br4wMAAACAXVh1yfWiRYu8br/00ktKSEjQ2rVr1a9fP1PyAADQUDhpCpQG1QBv1qyZsrOzvfbl5OQoKChIcXFxVd4nNDRUoaGh9RHvqJpGhSomLEh5h8u0dW+hOjaPsTQPAAAAADhdbm6uJKlJkyYWJwEAAHWpQTXAe/XqpQ8//NBr36effqoePXooODjYolTH5nK5lJIQpW8yDmpTTgENcAAAAACOVW4EqNykkWIV5/nzWk/HGghlGIYmTJigs846S507dzYlCwAADUld1GO7sjRdQUGB1q9fr/Xr10uStm7dqvXr1ysjI0PSkelLrrvuOs/xY8aM0fbt2zVhwgT99NNPmjt3rubMmaO77rrLivg+YSFMAAAAAJAMueQ2aTN0ZCrMpKQkr7Wf0tLSqs1wyy236LvvvtMbb7xRH08ZAADbqYt6bFeWjgBfs2aNBg4c6LldMVf39ddfr3nz5ikrK8vTDJek5ORkffTRR7rjjjv0r3/9Sy1atNCzzz6ryy67rN6z+yqFhTABAAAAoE5kZmYqJub3K22rG/1966236oMPPtCyZcvUsmXL+ogHAAAsZGkDfMCAAZ5FLKsyb968Svv69++vb775pg5T1Y2KBvhmRoADAAAAcLC6uOQ6JibGqwFeFcMwdOutt2rhwoVasmSJkpOTTckAAEBD5KQpUBrUHOANWUrTaEnSlr2FKncbCgyw96UBAAAAAFAX3IZLbsOcz0O+nGfcuHGaP3++3n//fUVHRys7O1uSFBsbq/DwcFPyAADQUFhVj61g7/a8HzmhcbhCgwJUUuZW5v4iq+MAAAAAgKPMnDlTubm5GjBggJo3b+7ZFixYYHU0AABQhxgBXk8CA1xq2zRKP2XlaVNOgdrER1odCQAAAADqXbkCVG7SWCxfzlPd9JsAADiNVfXYCvZO52dYCBMAAACA01Vccm3WBgAAfOekekwDvB6lNP2tAc5CmAAAAAAAAABQ55gCpR55RoDTAAcAAADgUG4FyG3SWCyzzgMAgNM4qR7bO52fqWiAb84pYP45AAAAAAAAAKhjjACvR23iIxTgkvKLy5STX6zEmDCrIwEAAABAvSo3XCo3aa5Qs84DAIDTOKke0wCvR6FBgWodF6mtewu1KaeABjgAAAAAxzFzsSy7L7oFAIBdOakeMwVKPWvHQpgAAAAAAAAAUC8YAV7PUhKi9L+fduvXnHyrowAAAABAvTOMALkNc8ZiGSadBwAAp3FSPaYBXs8qFsJkBDgAAAAAJyqXS+Uyac5Rk84DAIDTOKke27s974d+b4AXWpwEAAAAAAAAAPwbI8DrWbumkZKkvQXFyi0qVWxEsMWJAAAAAKD+uA3zFstyG6acBgAAx3FSPaYBXs+iw4LVLCZM2XmHtWlPvrq3bmJ1JAAALFNwSb4CI0qsjiFJOvxrrNURvOS1bml1BC/GbrfVEbyEHbDXhYx7X2tjdYRKWq/aYHUELxd8s9PqCF7eunew1RG89Lj0J6sjeJQUlOhNq0Og3sXdnKHgyBCrY0iSdv7Y2uoIXlzn77M6gpdte+3VR2jUIs/qCF5ee2+g1REqSX431+oIXlZ8NMPqCF6SPx5tdQQvwQH2+L3XZZMc/oAGuAVSEqKONMBzCmiAAwAAAHAUt4mLbpl1HgAAnMZJ9dje6fwUC2ECAAAAcCq3XKZuAADAd06qxzTALdCOBjgAAAAAAAAA1DmmQLHAiRUN8D00wAEAAAA4S7nhUrlJi26ZdR4AAJzGSfWYBrgFKqZA2XHgkA6XlissONDiRAAAAABQP5w05ygAAHblpHps73R+Ki4yRI0igmUY0mZGgQMAAAAAAABAnaABbgGXy6WUpswDDgAAAMB53HLJbZi02XzRLQAA7MpJ9ZgGuEUqpkHZTAMcAAAAAAAAAOoEc4BbJIWFMAEAAAA4kCHzRooZNh9xBgCAXTmpHtMAt0i7BKZAAQAAAOA8FZdLm3UuAADgOyfVY6ZAsUjFHOBb9xaqrNxtcRoAAAAAAAAA8D+MALfICY3CFR4cqEOl5crYX6S2vzXEAQAAAMCfuY0AuQ1zxmKZdR4AAJzGSfXY3un8WECAS22bRkpiGhQAAAAAzlFxybVZGwAA8J2T6jENcAuxECYAoKFKS0vT6aefrujoaCUkJOjiiy/Wxo0brY4FAICjUI8BAA3RsmXLNHToULVo0UIul0vvvfdenT4eDXALVcwDzghwAEBDs3TpUo0bN05ffvml0tPTVVZWptTUVBUWFlodDQBgc265TN2cjHoMAKgtK+txYWGhTj31VD3//PN19Oy8MQe4hSpGgG+mAQ4AaGAWLVrkdfull15SQkKC1q5dq379+lmUCgDQEJh5qbTdL7mua9RjAEBtWVmPBw8erMGDB5vy2DVBA9xCngb4nkIZhiGXy9m/vAEAGq7c3FxJUpMmTar8enFxsYqLiz238/Ly6iUXAABOcqx6LFGTAQB15881JTQ0VKGhoRal+R1ToFiodVykAgNcKiguU3beYavjAABQK4ZhaMKECTrrrLPUuXPnKo9JS0tTbGysZ0tKSqrnlAAAu3DSolv1qSb1WKImAwCOqIt6nJSU5FVj0tLSLH6WR9AAt1BIUIBax0VIYh5wAEDDdcstt+i7777TG2+8cdRjJk6cqNzcXM+WmZlZjwkBAPB/NanHEjUZAFB3MjMzvWrMxIkTrY4kiSlQLJfSNEpb9hRqU06B+p7Y1Oo4AAD45NZbb9UHH3ygZcuWqWXLlkc9zi6XvgEArMcc4OaraT2WqMkAgCPqoh7HxMQoJibGlHOaiQa4xVISovTpht2MAAcANCiGYejWW2/VwoULtWTJEiUnJ1sdCQDQQNAANw/1GABQW06qxzTALVaxECYNcABAQzJu3DjNnz9f77//vqKjo5WdnS1Jio2NVXh4uMXpAABwBuoxAKAhKigo0KZNmzy3t27dqvXr16tJkyZq1aqV6Y9HA9xiFQ3wzXtogAMAGo6ZM2dKkgYMGOC1/6WXXtLIkSPrPxAAoMEwJLllzkgxw5SzNFzUYwBAbVlZj9esWaOBAwd6bk+YMEGSdP3112vevHmmZPojGuAWa9f0SAN8b0GJDhaVqFFEiMWJAAA4NsNwessBAFBbTrrkuq5RjwEAtWVlPR4wYEC91rCAenskVCkyNEgtYsMkMQ0KAAAAAAAAAJiJBrgNtGMecAAAAAAOUTHizKwNAAD4zkn1mAa4DbAQJgAAAAAAAACYjznAbcDTAGchTAAAAAB+jjnAAQCwnpPqMQ1wG0hpyghwAAAAAM7gpA/cAADYlZPqseVToMyYMUPJyckKCwtT9+7dtXz58mqP/9e//qWOHTsqPDxc7du31yuvvFJPSetOxQjwHQcOqaikzOI0AAAAAAAAAOAfLG2AL1iwQOPHj9ekSZO0bt069e3bV4MHD1ZGRkaVx8+cOVMTJ07U5MmT9eOPP+rhhx/WuHHj9OGHH9ZzcnPFRYWqcUSwJGnLnkKL0wAAAABA3TEMl6kbAADwnZPqsaUN8GnTpmnUqFEaPXq0OnbsqOnTpyspKUkzZ86s8vhXX31VN998s4YPH662bdvqyiuv1KhRozR16tR6Tm4+FsIEAAAA4ARuuUzdAACA75xUjy1rgJeUlGjt2rVKTU312p+amqqVK1dWeZ/i4mKFhYV57QsPD9fXX3+t0tLSOstaH2iAAwAAAAAAAIC5LGuA7927V+Xl5UpMTPTan5iYqOzs7Crvc+655+rFF1/U2rVrZRiG1qxZo7lz56q0tFR79+6t8j7FxcXKy8vz2uyoHQthAgAAAHCAikW3zNoAAIDvnFSPLV8E0+XyfoEMw6i0r8IDDzygwYMH68wzz1RwcLAuuugijRw5UpIUGBhY5X3S0tIUGxvr2ZKSkkzNbxbPCPA9NMABAAAAAAAAwAxBVj1wfHy8AgMDK432zsnJqTQqvEJ4eLjmzp2rf//739q9e7eaN2+u2bNnKzo6WvHx8VXeZ+LEiZowYYLndl5eni2b4CcmRkuStu0tVGm5W8GBlv9tAgCAOtdofqSCgsOOfWA9yDyv3OoIXpJeOmB1BC8bb46zOoKXti9VvWi6VTb+s6nVESopv/IUqyN4+eWT06yO4MVINayO4GX/u/b5fpUXH67T85u5WJbdF91qSDIXJiswxB41WV3sNcVp8Yqq+w1WMU4rtDqCl9xtjayO4KVlz6pnFbDSzi42+bf1m1OeHmt1BC9BTe1Vk3cEN7Y6giTJfYh6bBbLuqwhISHq3r270tPTvfanp6erd+/e1d43ODhYLVu2VGBgoN58801dcMEFCgio+qmEhoYqJibGa7OjFrFhiggJVJnb0PZ9RVbHAQAAAIA64aRLrgEAsCsn1WPLRoBL0oQJEzRixAj16NFDvXr10uzZs5WRkaExY8ZIOjJ6e+fOnXrllVckSb/88ou+/vpr9ezZUwcOHNC0adP0ww8/6OWXX7byaZjC5XKpXdMofb8zV5tyCjxTogAAAAAAAAAAasfSBvjw4cO1b98+TZkyRVlZWercubM++ugjtW7dWpKUlZWljIzfL28tLy/XU089pY0bNyo4OFgDBw7UypUr1aZNG4uegblSEo40wDczDzgAAAAAP+WkS64BALArJ9VjSxvgkjR27FiNHVv13EPz5s3zut2xY0etW7euHlJZw7MQZg4NcAAAAAD+yTDxUmm7f+AGAMCunFSPWWnRRto1pQEOAAAAAAAAAGaxfAQ4flcxAnzzngK53YYCAuz91xMAAAAA8JUhyTDMOxcAAPCdk+oxDXAbaR0XoaAAl4pKypWVd1gnNAq3OhIAAAAAmMotl1wyZ7CP26TzAADgNE6qx0yBYiPBgQFqEx8piWlQAAAAAAAAAOB40QC3mRTmAQcAAADgxwzDZeoGAAB856R6TAPcZirmAacBDgAAAAAAAADHhznAbcazECYNcAAAAAB+yG245DJppJjb5iPOAACwKyfVYxrgNuMZAb6HBjgAAAAA/2MYRzazzgUAAHznpHrMFCg207bpkUUw9xeWaH9hicVpAAAAAAAAAKDhogFuMxEhQTqhUbgk5gEHAAAA4H+ctOgWAAB25aR6TAPchlgIEwAAAIC/ctIHbgAA7MpJ9ZgGuA3RAAcAAAAAAACA48cimDbEQpgAAAAA/JXbcMll0kgxt81HnAEAYFdOqseMALehigb4ZkaAAwAAAAAAAECtMQLchlKaHmmA7zx4SIXFZYoM5dsEAAAAwD8YxpHNrHMBAADfOakeMwLchhpHhiguMkSStGVPocVpAAAAAMA8Rz5wm7XoltXPBgCAhslJ9ZgGuE2188wDnm9xEgAAAADwD8uWLdPQoUPVokULuVwuvffee1ZHAgAAdYwGuE15FsJkHnAAAAAAfsS80WZHNl8UFhbq1FNP1fPPP19Hzw4AgIbBynpc35hc2qYq5gGnAQ4AAADAnxi/bWadyxeDBw/W4MGDTXp0AAAaLivrcX1jBLhNMQIcAAAAAAAAAI4PI8BtqqIBvn1fkUrL3QoO5G8VAAAAABo+My+VrjhPXl6e1/7Q0FCFhoaa8hgAAPijuqjHdkVX1aaax4YpMiRQZW5D2/cVWh0HAAAAAMxhmLxJSkpKUmxsrGdLS0urxycEAEADVAf12K4YAW5TLpdL7RKi9N2OXP26u0ApCdFWRwIAAAAAW8rMzFRMTIznNqO/AQBABRrgNpbS9EgDnHnAAQAAAPgNEy+51m/niYmJ8WqAAwCAY6iDemxXNMBtrF3FQph7aIADAAAAwPEqKCjQpk2bPLe3bt2q9evXq0mTJmrVqpWFyQAAQF2hAW5jFQthMgIcAAAAgL8wjCObWefyxZo1azRw4EDP7QkTJkiSrr/+es2bN8+cUAAANABW1uP6RgPcxk78rQG+eU+B3G5DAQH2vpwAAAAAAI7FMPGSa1/PM2DAABl2/5QOAEA9sLIe17cAqwPg6Fo1iVBIYIAOl7q18+Ahq+MAAAAAAAAAQINCA9zGggID1CY+QhLzgAMAAADwE4bL3A0AAPjOQfWYBrjNVcwDvpl5wAEAAAD4gYo5R83aAACA75xUj2mA21xKUxbCBAAAAAAAAIDaYBFMm2uXQAMcAODHRu2VIkOtTiFJajq/mdURvGy9oqnVEbwkv3/Y6gheitslWB3BS/u7dlkdoZJN45KtjuCl0UarE/yZvS7VbfTqSqsjeJQZparTb5fx22bWuWCKxhfsVJBNanLZopZWR/DW56DVCby0fdReYxmze9srT9Qs+7W6Mm+PtjqCl/h99vrhWdjGbXUELyc9U2x1BElSWXmJMuvyARxUj+31UwqVVEyBsmlPAauVAwAAAAAAAIAP7PdnMXhp1zRKLpd0sKhU+wpLFB9lj7/IAwAAAEBtGIZLhkmLZZl1HgAAnMZJ9ZgR4DYXFhyolo3DJTENCgAAAAA/YZi0AQCA2nNIPaYB3gCwECYAAAAAAAAA+I4GeAOQwkKYAAAAAPxExSXXZm0AAMB3TqrHPjfADx06pKKiIs/t7du3a/r06fr0009NDYbfVTTAN++hAQ4AAACggTPrcusGctk1AAC25KB67HMD/KKLLtIrr7wiSTp48KB69uypp556ShdddJFmzpxpekAwAhwAAAAAAAAAasPnBvg333yjvn37SpL+85//KDExUdu3b9crr7yiZ5991vSAkFKaRkuSsnIPq6C4zOI0AAAAAHA8XCZvAADAd86pxz43wIuKihQdfaQh++mnn+rSSy9VQECAzjzzTG3fvt30gJBiI4IVHxUqSdrMKHAAAAAAAAAAqBGfG+ApKSl67733lJmZqU8++USpqamSpJycHMXExJgeEEekJERKYhoUAAAAAA2cg+YcBQDAthxUj31ugD/44IO666671KZNG/Xs2VO9evWSdGQ0eNeuXU0PiCM884CzECYAAACAhsxBH7gBALAtB9XjIF/vcPnll+uss85SVlaWTj31VM/+QYMG6ZJLLjE1HH6X0pSFMAEAAAAAAADAFz43wCWpWbNmatasmSQpLy9PixcvVvv27dWhQwdTw+F3KQlH5l1nDnAAAAAADZrhOrKZdS4AAOA7B9Vjn6dAueKKK/T8889Lkg4dOqQePXroiiuuUJcuXfTOO+/4HGDGjBlKTk5WWFiYunfvruXLl1d7/Ouvv65TTz1VERERat68uW644Qbt27fP58dtaCqmQNm+v0glZW6L0wAA4C0tLU0ul0vjx4+3OgoAwOYMw9wNv6MeAwBqykn12OcG+LJly9S3b19J0sKFC2UYhg4ePKhnn31WjzzyiE/nWrBggcaPH69JkyZp3bp16tu3rwYPHqyMjIwqj1+xYoWuu+46jRo1Sj/++KPefvttrV69WqNHj/b1aTQ4iTGhigoNUrnb0LZ9hVbHAQDAY/Xq1Zo9e7a6dOlidRQAAByLegwAQNV8boDn5uaqSZMmkqRFixbpsssuU0REhM4//3z9+uuvPp1r2rRpGjVqlEaPHq2OHTtq+vTpSkpK0syZM6s8/ssvv1SbNm102223KTk5WWeddZZuvvlmrVmzxten0eC4XC61S2AecACAvRQUFOiaa67RCy+8oMaNG1sdBwDQEDho0a36Qj0GAPjMQfXY5wZ4UlKSVq1apcLCQi1atEipqamSpAMHDigsLKzG5ykpKdHatWs996+QmpqqlStXVnmf3r17a8eOHfroo49kGIZ2796t//znPzr//PN9fRoNEgthAgDsZty4cTr//PP1l7/8xeooAICGomLOUbM2UI8BAL5zUD32eRHM8ePH65prrlFUVJRat26tAQMGSDoyNcopp5xS4/Ps3btX5eXlSkxM9NqfmJio7OzsKu/Tu3dvvf766xo+fLgOHz6ssrIyXXjhhXruueeO+jjFxcUqLi723M7Ly6txRrtJYQQ4AMBG3nzzTX3zzTdavXr1MY/1p3oMAICd+FKPJWoyAMB5fB4BPnbsWK1atUpz587VihUrFBBw5BRt27b1eQ5w6cjUHn9kGEalfRU2bNig2267TQ8++KDWrl2rRYsWaevWrRozZsxRz5+WlqbY2FjPlpSU5HNGu6ABDgCwi8zMTN1+++167bXXanQFmD/VYwDA8XEZ5m5O5ms9lqjJAIAjnFSPfW6AS1KPHj10ySWXKDIyUsZvy3yef/756tOnT43PER8fr8DAwEqjvXNyciqNCq+QlpamPn366O6771aXLl107rnnasaMGZo7d66ysrKqvM/EiROVm5vr2TIzM2uc0W4qGuBb9hbI7bb5OwsA4NfWrl2rnJwcde/eXUFBQQoKCtLSpUv17LPPKigoSOXl5V7H+1M9BgDALnytxxI1GQDgPLVqgL/yyis65ZRTFB4ervDwcHXp0kWvvvqqT+cICQlR9+7dlZ6e7rU/PT1dvXv3rvI+RUVFnhHnFQIDAyXJ04j/s9DQUMXExHhtDVVS43CFBAbocKlbOw8esjoOAMDBBg0apO+//17r16/3bD169NA111yj9evXe+pzBX+qxwCA4+SgRbfqmq/1WKImAwB+46B67PMc4NOmTdMDDzygW265RX369JFhGPriiy80ZswY7d27V3fccUeNzzVhwgSNGDFCPXr0UK9evTR79mxlZGR4pjSZOHGidu7cqVdeeUWSNHToUN10002aOXOmzj33XGVlZWn8+PE644wz1KJFC1+fSoMTFBig5PhIbdydr005BUpqEmF1JACAQ0VHR6tz585e+yIjIxUXF1dpPwAAXsxcLMvmi27VNeoxAKDWHFSPfW6AP/fcc5o5c6auu+46z76LLrpInTp10uTJk31qgA8fPlz79u3TlClTlJWVpc6dO+ujjz5S69atJUlZWVnKyMjwHD9y5Ejl5+fr+eef15133qlGjRrp7LPP1tSpU319Gg1WSkKUpwE+sEOC1XEAAAAAAAAAwLZ8boBnZWVVOUVJ7969jzoPd3XGjh2rsWPHVvm1efPmVdp366236tZbb/X5cfxFOxbCBADY1JIlS6yOAABoCMy8VNrml1xbgXoMAKgRB9Vjn+cAT0lJ0VtvvVVp/4IFC3TiiSeaEgpHV7EQ5q85+RYnAQAAAIBacNCcowAA2JaD6rHPI8AffvhhDR8+XMuWLVOfPn3kcrm0YsUKffbZZ1U2xmGulKa/jwA3DEMul73n2AEAAAAAAAAAq/g8Avyyyy7TV199pfj4eL333nt69913FR8fr6+//lqXXHJJXWTEH7RtGqkAl5R3uEx7CoqtjgMAAAAAvnHQiDMAAGzLQfXY5xHgktS9e3e99tprXvt2796tKVOm6MEHHzQlGKoWFhyopCYR2r6vSJtyCpQQHWZ1JAAAAAAAAACwJZ9HgB9Ndna2Hn74YbNOh2pUTIOymYUwAQAAADQ0hsvcDQAA+M7iejxjxgwlJycrLCxM3bt31/Lly+vgSR5hWgMc9adiIcxNNMABAAAANDAuw9wNAAD4zsp6vGDBAo0fP16TJk3SunXr1LdvXw0ePFgZGRl18lxpgDdA7Soa4HtogAMAAAAAAABoOKZNm6ZRo0Zp9OjR6tixo6ZPn66kpCTNnDmzTh6PBngDxAhwAAAAAA2WgxbdAgDAtuqgHufl5XltxcXFlR62pKREa9euVWpqqtf+1NRUrVy50vSnKfmwCOaECROq/fqePXuOOwxqpqIBvjuvWHmHSxUTFmxxIgAAAAAAAABOlpSU5HX7oYce0uTJk7327d27V+Xl5UpMTPTan5iYqOzs7DrJVeMG+Lp16455TL9+/Y4rDGomJixYCdGhyskv1uacAnVt1djqSAAAAAAAAAAcLDMzUzExMZ7boaGhRz3W5fJeONMwjEr7zFLjBvjnn39eJwFQOykJUcrJL9YmGuAAAAAAGhCXzFu8sm4+JgMA4P/qoh7HxMR4NcCrEh8fr8DAwEqjvXNyciqNCjdLjRvgsJeUhCit3LyPhTABAA3ageXNFBgaZnUMSVJAvNUJvJVG2Wti26wz7fF9quDulm91BC/hobFWR6ik/Fd7vYfyW9tr+aGwvfZ6fRJXVf9hsT6VFpZIg6xOgfq2e+kJtqnJRSeXWB3BS+h3jayO4CX/kbqZIqC28rKPPsLTCnEXVp5z2GoBP9irBu4feNjqCF5ce+z1Hsp/9JDVESRJZYXF0qVWpzBfSEiIunfvrvT0dF1yySWe/enp6brooovq5DFpgDdQFfOAb2YhTAAAAAANieE6spl1LgAA4DsL6/GECRM0YsQI9ejRQ7169dLs2bOVkZGhMWPGmJPnT2iAN1ApTY80wDfRAAcAAADQkBi/bWadCwAA+M7Cejx8+HDt27dPU6ZMUVZWljp37qyPPvpIrVu3NimQNxrgDVTFCPCM/UU6XFqusOBAixMBAAAAAAAAwLGNHTtWY8eOrZfHstckRKixptGhig4LktuQtu0rtDoOAAAAANSMYfIGAAB856B6XOMG+OOPP65Dh36fBH7ZsmUqLv59YYH8/Px669pDcrlcnlHgTIMCAAAAoKFwGeZuAADAd06qxzVugE+cOFH5+fme2xdccIF27tzpuV1UVKR///vf5qZDtZgHHAAAAAAAAACOrsZzgBuGUe1t1D9GgAMAAABocFgEEwAA6zmoHrMIZgNGAxwAAABAg+OgD9wAANiWg+oxi2A2YBUN8C17C1Xutvk7DQAAAAAAAADqmU8jwF988UVFRR1pupaVlWnevHmKj4+XJK/5wVE/WjaOUEhQgErK3NpxoEit4yKtjgQAAAAA1TJzsSy7L7oFAIBdOake17gB3qpVK73wwgue282aNdOrr75a6RjUn8AAl9rGR+rn7HxtyimgAQ4AAAAAAAAAf1DjBvi2bdvqMAZqKyUhytMAH9Qx0eo4AAAAAFA9w3VkM+tcAADAdw6qxyyC2cCxECYAAACABsVBi24BAGBbDqrHNV4E86uvvtLHH3/ste+VV15RcnKyEhIS9Ne//lXFxcWmB0T1PA3wPTTAAQAAAAAAAOCPatwAnzx5sr777jvP7e+//16jRo3SX/7yF91333368MMPlZaWVichcXR/HAFuGDb/cwsAAAAAx6tYdMusDQAA+M5J9bjGDfD169dr0KBBnttvvvmmevbsqRdeeEETJkzQs88+q7feeqtOQuLokuMjFeCS8g+XaU8+I/ABAAAA2Jxh8gYAAHznoHpc4wb4gQMHlJj4+yKLS5cu1Xnnnee5ffrppyszM9PcdDim0KBAtWoSIYl5wAEAAAAAAADgj2rcAE9MTNTWrVslSSUlJfrmm2/Uq1cvz9fz8/MVHBxsfkIcE/OAAwAAAGgwzLzc2uYjzgAAsC0H1eMaN8DPO+883XfffVq+fLkmTpyoiIgI9e3b1/P17777Tu3atauTkKheuz/MAw4AAAAAAAAAOCKopgc+8sgjuvTSS9W/f39FRUXp5ZdfVkhIiOfrc+fOVWpqap2ERPVSmtIABwAAANBAmDlSzOYjzgAAsC0H1eMaN8CbNm2q5cuXKzc3V1FRUQoMDPT6+ttvv62oqCjTA+LYUhgBDgAAAKChcNAHbgAAbMtB9bjGU6BUiI2NrdT8lqQmTZp4jQhH/alogOfkFyv3UKnFaQAAAAAAAADAHmo8AvzGG2+s0XFz586tdRjUTnRYsJrFhCk777A25RSoe+vGVkcCAAAAgCp5Fswy6VwAAMB3TqrHNW6Az5s3T61bt1bXrl1lGDZ/Vg6UkhCl7LzD2kwDHAAAAAAAAAAk+dAAHzNmjN58801t2bJFN954o6699lo1adKkLrPBBykJUVqxaa827WEecAAAAAAAAACQfJgDfMaMGcrKytK9996rDz/8UElJSbriiiv0ySefMCLcBtqxECYAAACAhsAweQMAAL5zUD32aRHM0NBQXXXVVUpPT9eGDRvUqVMnjR07Vq1bt1ZBAY1XK6U0pQEOAAAAAAAAAH9U4ylQ/szlcsnlcskwDLndbjMzoRZSfhsBnnmgSIdLyxUWHGhxIgAAAACozEmLbgEAYFdOqsc+jQAvLi7WG2+8oXPOOUft27fX999/r+eff14ZGRmKioqqq4yogfioEMWGB8swpC17Cq2OAwAAAABHZ+Hl1jNmzFBycrLCwsLUvXt3LV++/LieCgAADZYDpj+RfGiAjx07Vs2bN9fUqVN1wQUXaMeOHXr77bc1ZMgQBQT41EdHHXC5XJ5R4CyECQAAAACVLViwQOPHj9ekSZO0bt069e3bV4MHD1ZGRobV0QAAQB2p8RQos2bNUqtWrZScnKylS5dq6dKlVR737rvvmhYOvklpGqW12w8wDzgAAAAA+zJztJiP55k2bZpGjRql0aNHS5KmT5+uTz75RDNnzlRaWppJoQAAaAAsrMf1rcYN8Ouuu04ul6sus+A4VYwA30wDHAAAAIBN1cWco3l5eV77Q0NDFRoa6rWvpKREa9eu1X333ee1PzU1VStXrjQnEAAADYST5gCvcQN83rx5dRgDZvBMgUIDHAAAAICDJCUled1+6KGHNHnyZK99e/fuVXl5uRITE732JyYmKjs7u64jAgAAi1g+ebcvC5CMHDlSLper0tapU6d6TGxfFQ3wrXsLVVbutjgNAAAAAFTBrAW3/nDpdmZmpnJzcz3bxIkTj/rwf76y2TAMrnYGADhPHdRju7K0Ae7rAiTPPPOMsrKyPFtmZqaaNGmiYcOG1XNyezqhUbjCggNUUu5W5oFDVscBAAAAgEoqLrk2a5OkmJgYr+3P059IUnx8vAIDAyuN9s7Jyak0KhwAAH9XF/XYrixtgP9xAZKOHTtq+vTpSkpK0syZM6s8PjY2Vs2aNfNsa9as0YEDB3TDDTfUc3J7CghwqW0806AAAAAAwJ+FhISoe/fuSk9P99qfnp6u3r17W5QKAADUtRrPAW42MxYgmTNnjv7yl7+odevWRz2muLhYxcXFntt/XhzF36QkRGlDVp425RTonJMZxQAAsLnueVJE8bGPqwf5OZFWR/ASnmnZr2lVKom117AOY7u9vl8JS8OtjlBJ8SmWz3bopfHGcqsjeCmNsNfrs/znE62O4OE+dLhuH8DMS6V9PM+ECRM0YsQI9ejRQ7169dLs2bOVkZGhMWPGmBSo4TqUVKaA8DKrY0iSAg7aqwYWN7XH61Ih51t7fdZ3hdjrd4TMrBOsjlCJy15vablzQ6yO4C3IXu+hnO/s8W/Mfdh/63F9s+yf4PEuQJKVlaWPP/5Y8+fPr/a4tLQ0Pfzww8eVtSFhIUwAAAAAqNrw4cO1b98+TZkyRVlZWercubM++uijagdVAQCAhs3yYQ+1XYBk3rx5atSokS6++OJqj5s4caLXYiiZmZnHE9f2PA3wPTTAAQAAANiQxYtujR07Vtu2bVNxcbHWrl2rfv36HecTAgCgAXLQIpiWjQA/ngVIDMPQ3LlzNWLECIWEVH/ZRmhoaJULoPirigb45pwCVjMHAAAAYDtmLpZl90W3AACwKyfVY8tGgB/PAiRLly7Vpk2bNGrUqLqM2CC1iYtUYIBLBcVl2p1njzlVAQAAAAAAAMAKlk6BMmHCBL344ouaO3eufvrpJ91xxx1eC5BMnDhR1113XaX7zZkzRz179lTnzp3rO7LthQQFqHWTCEnMAw4AAADAhhx0yTUAALbloHps6Tq0x1qAJCsrSxkZGV73yc3N1TvvvKNnnnnGisgNQruEKG3ZW6hNOfk668R4q+MAAAAAwO/M/KBs8w/cAADYloPqsaUNcOnIAiRjx46t8mvz5s2rtC82NlZFRUV1nKphS0mIUvqG3SyECQAAAAAAAMDRLG+Aw3wpTY8shMkUKAAAAADsxkmLbgEAYFdOqseWzgGOupGSUNEAL7Q4CQAAAAAAAABYhxHgfqjdbw3wvQXFyi0qVWxEsMWJAAAAAOA3DppzFAAA23JQPWYEuB+KCg1S89gwSdKmPfkWpwEAAACA31Vccm3WBgAAfOekekwD3E/9Pg0K84ADAAAAAAAAcCYa4H6qHQthAgDq2M6dO3XttdcqLi5OEREROu2007R27VqrYwEA7M4weXM46jEAoFYcVI+ZA9xPMQIcAFCXDhw4oD59+mjgwIH6+OOPlZCQoM2bN6tRo0ZWRwMA2J2D5hyta9RjAECtOage0wD3U54G+B4a4AAA802dOlVJSUl66aWXPPvatGljXSAAAByIegwAwLExBYqfqmiA7zhwSIdLyy1OAwDwNx988IF69OihYcOGKSEhQV27dtULL7xgdSwAQAPgMnlzMuoxAKC2nFSPaYD7qbjIEDWOCJZhSJsZBQ4AMNmWLVs0c+ZMnXjiifrkk080ZswY3XbbbXrllVeqPL64uFh5eXleGwAAOD6+1mOJmgwAcB4a4H7K5XIxDzgAoM643W5169ZNjz32mLp27aqbb75ZN910k2bOnFnl8WlpaYqNjfVsSUlJ9ZwYAGAbDlp0q675Wo8lajIA4DcOqsc0wP1YRQN8Mw1wAIDJmjdvrpNPPtlrX8eOHZWRkVHl8RMnTlRubq5ny8zMrI+YAAAbchnmbk7maz2WqMkAgCOcVI9ZBNOPtWt6pAH+Kw1wAIDJ+vTpo40bN3rt++WXX9S6desqjw8NDVVoaGh9RAMAwDF8rccSNRkA4DyMAPdjTIECAKgrd9xxh7788ks99thj2rRpk+bPn6/Zs2dr3LhxVkcDANidgy65rmvUYwBArTmoHtMA92MVDfBt+wpVVu62OA0AwJ+cfvrpWrhwod544w117txZ//jHPzR9+nRdc801VkcDADQEDviwXR+oxwCA4+KQeswUKH6sRWy4woMDdai0XNv3F3mmRAEAwAwXXHCBLrjgAqtjAADgaNRjAACqxwhwPxYQ4FK7hEhJTIMCAAAAwB6ctOgWAAB25aR6TAPcz6U0ZR5wAAAAADbioDlHAQCwLQfVYxrgfq5iHvDNNMABAAAAAAAAOAxzgPu5igb4pj00wAEAAABYz8xLpe1+yTUAAHblpHrMCHA/98cR4IZh83cjAAAAAAAAAJiIBrifax0XqaAAlwpLypWVe9jqOAAAAACczkFzjgIAYFsOqsc0wP1ccGCAWsdFSGIhTAAAAADWq7jk2qwNAAD4zkn1mAa4A3jmAacBDgAAAAAAAMBBaIA7AAthAgAAALANB11yDQCAbTmoHgdZHQB1jxHgAAAAAGzDzA/KNv/ADQCAbTmoHjMC3AFSmkZLkjbTAAcAAAAAAADgIIwAd4B2CZGSpH2FJTpQWKLGkSEWJwIAAADgVGYulmX3RbcAALArJ9VjGuAOEBESpBMahWvnwUPatKdAp0c2sToSAACSpJDPoxUYEmZ1DElS9EF7/dYWfKjM6gheIrfkWR3BS/6JsVZH8BK5cZ/VESq57al0qyN4mdP7dKsjeNl7/klWR/Bywv/Z56NZWWmQdlgdAvUuLCtIgaH2eR/aSXmoy+oIXpp9WW51BC9FCYFWR/BSFm6v75ckjfjbIqsjePmkc4zVEbxs/WcvqyN4afNBkdURJEllZYe1zeoQfoIpUByiHfOAAwAAALADBy26BQCAbTmoHtMAd4iUpjTAAQAAAFjPZRimbgAAwHdOqsc0wB0ihRHgAAAAAAAAAByGCb4cggY4AAAAAFsw81Jpew84AwDAvhxUj2mAO0RFA3znwUMqKilTRAjfegAAAAD1z2Uc2cw6FwAA8J2T6jFToDhEk8gQNYkMkSRt2VNocRoAAAAAAAAAqHs0wB2EhTABAAAAWM4weQMAAL5rIPX40UcfVe/evRUREaFGjRrV6hw0wB2kHfOAAwAAALBYxSXXZm0AAMB3DaUel5SUaNiwYfrb3/5W63MwEbSDsBAmAAAAAAAAgIbi4YcfliTNmzev1uegAe4gngb4HhrgAAAAACxi5qXSjAAHAKB26qAe5+Xlee0ODQ1VaGioSQ9Se0yB4iAVDfBtewtVWu62OA0AAAAAAAAAf5GUlKTY2FjPlpaWZnUkSTTAHaVFbJgiQwJV5ja0fV+R1XEAAAAAOFBDmXMUAAB/Vhf1ODMzU7m5uZ5t4sSJVT725MmT5XK5qt3WrFlj2nNlChQHcblcapcQpe925GpTToFnRDgAAAAA1BumQAEAwHp1UI9jYmIUExNzzMNvueUWXXnlldUe06ZNGxOCHUED3GFSmh5pgG9mHnAAAAAAAAAA9Sw+Pl7x8fH19ng0wB2mXcVCmDk0wAEAAABYg6lLAACwXkOoxxkZGdq/f78yMjJUXl6u9evXS5JSUlIUFVWz2S0snwN8xowZSk5OVlhYmLp3767ly5dXe3xxcbEmTZqk1q1bKzQ0VO3atdPcuXPrKW3Dl0IDHAAAAICVDMPcDQAA+K6B1OMHH3xQXbt21UMPPaSCggJ17dpVXbt29WmOcEtHgC9YsEDjx4/XjBkz1KdPH/373//W4MGDtWHDBrVq1arK+1xxxRXavXu35syZo5SUFOXk5KisrKyekzdcFQ3wzXsK5HYbCghwWZwIAAAAAAAAACqbN2+e5s2bd1znsLQBPm3aNI0aNUqjR4+WJE2fPl2ffPKJZs6cqbS0tErHL1q0SEuXLtWWLVvUpEkTSeZOiO4ErZtEKDjQpaKScu3KPaSWjSOsjgQAAADAQVyGeZdcN4RLtwEAsCMn1WPLpkApKSnR2rVrlZqa6rU/NTVVK1eurPI+H3zwgXr06KHHH39cJ5xwgk466STdddddOnToUH1E9gtBgQFqExcpiWlQAAAAAAAAAPg3y0aA7927V+Xl5UpMTPTan5iYqOzs7Crvs2XLFq1YsUJhYWFauHCh9u7dq7Fjx2r//v1HnQe8uLhYxcXFntt5eXnmPYkGKiUhSr/mFGhTToEGtE+wOg4AAAAAJzF+28w6FwAA8J2D6rHli2C6XN5zUBuGUWlfBbfbLZfLpddff11nnHGGhgwZomnTpmnevHlHHQWelpam2NhYz5aUlGT6c2ho/jgPOAAAAADUJ5fb3A0AAPjOSfXYsgZ4fHy8AgMDK432zsnJqTQqvELz5s11wgknKDY21rOvY8eOMgxDO3bsqPI+EydOVG5urmfLzMw070k0UBUNcKZAAQAAAAAAAODPLGuAh4SEqHv37kpPT/fan56ert69e1d5nz59+mjXrl0qKPi9cfvLL78oICBALVu2rPI+oaGhiomJ8dqcrl1TGuAAAAAALGKYvAEAAN85qB5bOgXKhAkT9OKLL2ru3Ln66aefdMcddygjI0NjxoyRdGT09nXXXec5/uqrr1ZcXJxuuOEGbdiwQcuWLdPdd9+tG2+8UeHh4VY9jQanXdMouVzSgaJS7SsoPvYdAAAAAMAkLsPcDQAA+M5J9diyRTAlafjw4dq3b5+mTJmirKwsde7cWR999JFat24tScrKylJGRobn+KioKKWnp+vWW29Vjx49FBcXpyuuuEKPPPKIVU+hQQoPCdQJjcK148AhbcopUFxUqNWRAAAAAAAAAMB0ljbAJWns2LEaO3ZslV+bN29epX0dOnSoNG0KfHdiQpR2HDik1dv2q2fbOKvjAAAAAHAKwziymXUuAADgOwfVY0unQIF1zu/SQpI0e9kWHSwqsTgNAAAAAAAAAJiPBrhDXdL1BHVoFq28w2WasWSz1XEAAAAAOIST5hwFAMCunFSPaYA7VGCAS/cO7iBJmvfFNu04UGRxIgAAAACOYJi8AQAA3zmoHtMAd7ABJzVVr7ZxKil3a9qnv1gdBwAAAAAAAABMRQPcwVwulyYOOTIKfOH6nfpxV67FiQAAAAD4Oyddcg0AgF05qR7TAHe4Li0baeipLWQY0j8//tnqOAAAAAD8nWGYuwEAAN85qB7TAIfuTm2v4ECXlv+6V8t/3WN1HAAAAAAAAAAwBQ1wqFVchK49s7WkI6PA3W57/9UGAAAAQMPlpEuuAQCwKyfVYxrgkCTdevaJig4N0o+78vTBt7usjgMAAADAXxkmb3Xk0UcfVe/evRUREaFGjRrV3QMBAGCFBlKPzRBkdQDYQ5PIEI0Z0E5PfLJRT3yyUed1bqaw4ECrYwEA/NyheJcCQ11Wx5AkBRZbncDb3qElVkfwctLfi6yO4GXX0EirI3jJb5lodYRK5lx8ntURvPz0zxirI3gJ2W11Am/NV7qtjvC7UhtlsVBJSYmGDRumXr16ac6cOVbHqXOt+21XUGSo1TEkSZtXtrY6gpeIzgesjuDl4L4mVkfwUtDKXj8zEr+yXyfutX+da3UEbx/stzqBl9Jd5VZH8LKzvz1+zywvDpS+sjqFf2AEODxu7JOsZjFh2nnwkF77crvVcQAAAAD4oYZyyfXDDz+sO+64Q6ecckrdPQgAABZpKPXYDDTA4REeEqgJ55wkSXpu8SblFpVanAgAAAAAAAAAao8GOLxc1r2lTkqMUu6hUs1YusnqOAAAAAD8jdswd5OUl5fntRUX22xeKwAA7KYO6rFd0QCHl8AAl+4b3EGS9NIX27Tz4CGLEwEAAADwK3Ww6FZSUpJiY2M9W1paWpUPPXnyZLlcrmq3NWvW1NUzBwDAPlgEE042sH2CeiY30Vdb92vap7/oqStOtToSAAAAABxVZmamYmJ+X2g1NLTqxRxvueUWXXnlldWeq02bNmZGAwAAFqMBjkpcLpcmDumoi//1hd5dt0Oj+yarY/OYY98RAAAAAI7BJfMWy3L99v8xMTFeDfCjiY+PV3x8vDkPDgBAA1YX9diumAIFVTotqZHO79JchiH98+OfrY4DAAAAwF8YhrlbHcnIyND69euVkZGh8vJyrV+/XuvXr1dBQUGdPSYAAPWmgdRjM9AAx1HdndpeQQEuLf1lj1Zu2mt1HAAAAACoNw8++KC6du2qhx56SAUFBeratau6du3KHOEAADQwNMBxVG3iI3Xtma0lSWkf/yy3zVd0BQAAAGB/LsPcra7MmzdPhmFU2gYMGFB3DwoAQD1pKPXYDDTAUa1bz05RVGiQvt+Zqw+/22V1HAAAAAAAAACoMRrgqFZcVKhu7tdWkvTkpxtVXFZucSIAAAAADZph8gYAAHznoHpMAxzHNKpvshKiQ5W5/5Be+zLD6jgAAAAAGjCXYZi6AQAA3zmpHtMAxzFFhATpjnNOkiQ9v/hX5R0utTgRAMBqZWVluv/++5WcnKzw8HC1bdtWU6ZMkdvttjoaAACOQT0GAODYgqwOgIZhWPeWmrNiqzblFGjWks2657wOVkcCAFho6tSpmjVrll5++WV16tRJa9as0Q033KDY2FjdfvvtVscDANiZ+7fNrHM5GPUYAFBrDqrHNMBRI0GBAbr3vA666ZU1mrNiq0b0aq3mseFWxwIAWGTVqlW66KKLdP7550uS2rRpozfeeENr1qyxOBkAwO7MvFTa7pdc1zXqMQCgtpxUj5kCBTX2l44JOqNNExWXufV0+i9WxwEAWOiss87SZ599pl9+OVIPvv32W61YsUJDhgyxOBkAAM5BPQYA4NgYAY4ac7lcum9IB106Y6X+s3aHRp3VVu2bRVsdCwBggXvvvVe5ubnq0KGDAgMDVV5erkcffVRXXXVVlccXFxeruLjYczsvL6++ogIA7Mb4bTPrXA7maz2WqMkAgN84qB4zAhw+6daqsQZ3bia3IU1d9LPVcQAAFlmwYIFee+01zZ8/X998841efvllPfnkk3r55ZerPD4tLU2xsbGeLSkpqZ4TAwDgf3ytxxI1GQDgPDTA4bO7z22voACXFv+co1Wb91kdBwBggbvvvlv33XefrrzySp1yyikaMWKE7rjjDqWlpVV5/MSJE5Wbm+vZMjMz6zkxAMA2DMPczcF8rccSNRkA8BsH1WMa4PBZ26ZRuuqMVpKkf378kwybv8kBAOYrKipSQID3rxGBgYFyu6te/js0NFQxMTFeGwDAmVyGuZuT+VqPJWoyAOAIJ9Vj5gBHrdw26ES9+80OfbsjV//9PksXdGlhdSQAQD0aOnSoHn30UbVq1UqdOnXSunXrNG3aNN14441WRwMAwDGoxwAAHBsNcNRK0+hQ/bVfOz39v1/0+KKNSj25mUKCuKAAAJziueee0wMPPKCxY8cqJydHLVq00M0336wHH3zQ6mgAALsz81Jph1+NSj0GANSag+oxDXDU2ui+yXrtq+3K2F+k+V9t18g+yVZHAgDUk+joaE2fPl3Tp0+3OgoAoIFxuY9sZp3LyajHAIDaclI9Zsguai0yNEjj/3KiJOnZxZuUf7jU4kQAAAAAAAAA8Dsa4Dguw3skqW3TSO0vLNG/l26xOg4AAAAAu6u45NqsDQAA+M5B9ZgGOI5LUGCA7j2vgyTpxRVblJ172OJEAAAAAGzNMHkDAAC+c1A9pgGO45Z6cqK6t26sw6VuTf/fL1bHAQAAAAAAAABJNMBhApfLpb8POTIK/K01mfp1d77FiQAAAADYlcswTN0AAIDvnFSPaYDDFN1bN9G5nRLlNqSpi362Og4AAAAAAAAA0ACHee45r4MCA1z63085+mrLPqvjAAAAALAjBy26BQCAbTmoHtMAh2naNY3SlacnSZLSPv5Zhs3f/AAAAAAsYEhym7TxkQMAgNpxUD2mAQ5T3f6XExUREqj1mQf18Q/ZVscBAAAAAAAA4GA0wGGqhOgw3dS3rSTp8UU/q7TcbXEiAAAAAHbipEW3AACwKyfVYxrgMN1N/doqPipE2/YV6Y2vM6yOAwAAAMBODJk456jVTwYAgAbKQfWYBjhMFxUapNv/cpIk6Zn//aqC4jKLEwEAAAAAAABwoiCrA8yYMUNPPPGEsrKy1KlTJ02fPl19+/at8tglS5Zo4MCBlfb/9NNP6tChQ11HhQ+uPD1JL63Yqi17CzV76WZNSG1vdSQAgA2VNHYrINwe02XFbbDXsIWyvBCrI3iz2bRmYb+GWh3BS367cqsjVLLLiLc6gpcOd3xvdQQvzdKtTuBteWgnqyN4uA8HSp/U4QNUjBYz61wwxS8bWiogPMzqGJKkIJsN1cvNiLU6gpdom73tQw7Y6xuW38rqBJUdPq3I6gheYv7bxOoIXgI62OtNHZJndYIjykvq+AEcVI8t/Sm1YMECjR8/XpMmTdK6devUt29fDR48WBkZ1U+bsXHjRmVlZXm2E088sZ4So6aCAwN097lHmt4vLN+qnLzDFicCAAAAAAAA4DSWNsCnTZumUaNGafTo0erYsaOmT5+upKQkzZw5s9r7JSQkqFmzZp4tMDCwnhLDF+d1bqaurRrpUGm5nv7fr1bHAQAAAGAHbpM3AADgOwfVY8sa4CUlJVq7dq1SU1O99qempmrlypXV3rdr165q3ry5Bg0apM8//7wuY+I4uFwu/X1IR0nSW2sytSmnwOJEAAAAAKzmMgxTNwAA4Dsn1WPLGuB79+5VeXm5EhMTvfYnJiYqOzu7yvs0b95cs2fP1jvvvKN3331X7du316BBg7Rs2bKjPk5xcbHy8vK8NtSf09s00TknJ6rcbejxRT9bHQcAAAAAAACAg1i+CKbL5fK6bRhGpX0V2rdvr/btf19MsVevXsrMzNSTTz6pfv36VXmftLQ0Pfzww+YFhs/uPa+9Pvtptz7dsFtrtu1Xjzb2WmwBAAAAQD1y0KJbAADYloPqsWUjwOPj4xUYGFhptHdOTk6lUeHVOfPMM/Xrr0efX3rixInKzc31bJmZmbXOjNpJSYjW8NOTJEmPffSTDJv/owAAAABQhyo+cJu1AQAA3zmoHlvWAA8JCVH37t2Vnp7utT89PV29e/eu8XnWrVun5s2bH/XroaGhiomJ8dpQ/8b/5SSFBwfqm4yD+uTHqqe4AQAAAAAAAAAzWToFyoQJEzRixAj16NFDvXr10uzZs5WRkaExY8ZIOjJ6e+fOnXrllVckSdOnT1ebNm3UqVMnlZSU6LXXXtM777yjd955x8qngRpIjAnT6L7Jem7xJj2+aKMGdUxUcKBlf38BAAAAYBUHXXINAIBtOageW9oAHz58uPbt26cpU6YoKytLnTt31kcffaTWrVtLkrKyspSRkeE5vqSkRHfddZd27typ8PBwderUSf/97381ZMgQq54CfPDXfm01/6sMbdlbqDdXZ2rEma2tjgQAAAAAAADAj1m+CObYsWM1duzYKr82b948r9v33HOP7rnnnnpIhboQHRas2wadqIc++FHP/O9XXdr1BEWGWv4WBAAAAFCf3JJcJp4LAAD4zkH1mDkoUK+uOqOV2sRFaG9BsV5YvsXqOAAAAADqmcswTN0AAIDvnFSPaYCjXoUEBejucztIkmYv26Kc/MMWJwIAAAAAAADgr2iAo94NOaWZTk1qpKKScj372a9WxwEAAABQnyoW3TJrAwAAvnNQPaYBjnrncrk0cfCRUeBvfJ2pzXsKLE4EAAAAoN64DXM3AADgOwfVYxrgsMSZbeM0qEOCyt2Gnli00eo4AAAAAAAAAPwQDXBY5t7BHRTgkhb9mK212w9YHQcAAABAfXDQJdcAANiWg+oxDXBY5qTEaA3rniRJSvvoJxk2/8cCAAAAwAxmftjmMwQAALXjnHpMAxyWuuOckxQWHKA12w8ofcNuq+MAAAAAAAAA8CM0wGGpZrFhGnVWsiRp6qKfVVbutjgRAAAAgDrloEuuAQCwLQfVYxrgsNzN/dupcUSwNu8p1FtrdlgdBwAAAAAAAICfoAEOy8WEBeu2QSdKkp7+3y8qKimzOBEAAACAOuM2zN0AAIDvHFSPaYDDFq7p2VqtmkRoT36xXly+1eo4AAAAAOqK4TZ3AwAAvmsA9Xjbtm0aNWqUkpOTFR4ernbt2umhhx5SSUmJT+ehAQ5bCAkK0N3ntpck/XvpZu0tKLY4EQAAAAAAAACr/Pzzz3K73fr3v/+tH3/8UU8//bRmzZqlv//97z6dhwY4bOP8U5qrS8tYFZaU69nPfrU6DgAAAIC64KBFtwAAsK0GUI/PO+88vfTSS0pNTVXbtm114YUX6q677tK7777r03logMM2AgJcum9wB0nS/K8ytHVvocWJAAAAAJjOQXOOAgBgW3VQj/Py8ry24mLzZ3jIzc1VkyZNfLoPDXDYSu928RrYvqnK3Iae+ORnq+MAAAAAAAAAqIGkpCTFxsZ6trS0NFPPv3nzZj333HMaM2aMT/ejAQ7buXdwB7lc0kffZ2tdxgGr4wAAAAAwUwO45BoAAL9XB/U4MzNTubm5nm3ixIlVPvTkyZPlcrmq3dasWeN1n127dum8887TsGHDNHr0aJ+ealDtXiGg7nRoFqPLu7XU22t3KO3jn7Xgr2fK5XJZHQsAAAAAAADAUcTExCgmJuaYx91yyy268sorqz2mTZs2nv/etWuXBg4cqF69emn27Nk+56IBDluakHqSPvh2l77eul+f/ZSjv5ycaHUkAAAAAGYwZN7IbQaAAwBQOxbW4/j4eMXHx9fo2J07d2rgwIHq3r27XnrpJQUE+D6hCVOgwJaax4brhj7JkqSpi35WWbnb4kQAAAAATMEUKAAAWK8B1ONdu3ZpwIABSkpK0pNPPqk9e/YoOztb2dnZPp2HBjhs628D2qlRRLB+zSnQf9busDoOAAAAAAAAgHry6aefatOmTVq8eLFatmyp5s2bezZf0ACHbcWGB+uWgSmSpKf/94uKSsosTgQAAADguLnd5m4AAMB3DaAejxw5UoZhVLn5ggY4bG1Er9Zq2Thcu/OKNXfFVqvjAAAAADheDeCSawAA/J6D6jENcNhaaFCg7j63vSRp1tIt2ldQbHEiAAAAAAAAAA1FkNUBgGMZ2qWFXli+RT/szNNzizdp8oWdrI4EADhOFZesuQ8ftjjJ78pK7TVqwX3IXnnK3Pb6I3R5sX3eO5LkPlRudYRKyosDrY7gpcwosTqCl5ICqxN4s9PPw4osvl5eXGNmjhSz+YizhsCONdl92F5j9dyB9ppqx24/38vt9SuCXDbLI0nuIvv8+5Kk8hKX1RG8uA/b62d5eYk9fgaVl1CPzUIDHLYXEODSxMEddc2LX+n1r7brhj5t1Dou0upYAIDjkJ+fL0na+dAjFif5XabVAf7sPasDeLPdctRPWB0AvtpgdYA/G2R1APvLz89XbGys+Sd2G5JM+qDstvcH7obAU5MfeNTiJACAqlCPjx8NcDQIfVLi1e+kplr2yx5dO+crtYmLVERIoCJCgn77/0CF//G/gwMVGRqk8JBARQQfOS78t69F/vbfIUH2+IseADhRixYtlJmZqejoaLlctR+BkpeXp6SkJGVmZiomJsbEhP6D16h6vD7V4/Wpnr++PoZhKD8/Xy1atLA6CuoBNbl+8PpUj9enerw+1fPX14d6bB4a4Ggw7juvg1Zu2qvM/YeUuf/QcZ8vKMDlaYpHhAT91jT/rZEeXNFUDzxqoz38t2a613HBNNcBoCYCAgLUsmVL084XExPjV7/s1gVeo+rx+lSP16d6/vj61MlIs98YhluGYc6UEmadx8moyfWL16d6vD7V4/Wpnj++PtRjc9AAR4NxcosY/fe2vtqyp0BFJeUqKin77f/Ldaj099uHSspVWFKuQ3+4/cfjy367LKPMbSj/cJnyD5dJMneSsKAAl1fjPDQ4UAHHMcXWcQzEkEu1v/PxPC4Ac406K1kXnXaC1TEAAAAAAGhQaICjQWnfLFrtm0Uf1zlKytxHmuKlVTfIi/7QPP9zc72ouFxFpTVrrucdLlPe4TIznjYAaG+BvRaPAwCg1gzDvLlCbb7oFgAAtuWgekwDHI4TEhSgkKAAxSrY9HMfrbl+qLS8xufwdXVfn4724WDDrIUQAJjixITj++OfvwoNDdVDDz2k0NBQq6PYFq9R9Xh9qsfrUz1en1oyTFx0y+YfuJ2Efw/V4/WpHq9P9Xh9qsfrU0sOqscuw9duWwOXl5en2NhY5ebm+t28QAAA1AS1EABghYr6Myh2hIJcIaacs8wo0We5r5pe07Zt26Z//OMfWrx4sbKzs9WiRQtde+21mjRpkkJCzMkOAIAVGlI9NgsjwAEAAAAA9cftllwmLZZVR4tu/fzzz3K73fr3v/+tlJQU/fDDD7rppptUWFioJ598sk4eEwCAetUA6rFZaIADAAAAAOpPA7jk+rzzztN5553nud22bVtt3LhRM2fOpAEOAPAPDaAemyXA6gAAAAAAANhdbm6umjRpYnUMAADgIxrgAACgwZoxY4aSk5MVFham7t27a/ny5VZHsoW0tDSdfvrpio6OVkJCgi6++GJt3LjR6li2lZaWJpfLpfHjx1sdxTZ27typa6+9VnFxcYqIiNBpp52mtWvXWh3LFsrKynT//fcrOTlZ4eHhatu2raZMmSK3296X/tqJ4XabuklH5jP941ZcXGxq5s2bN+u5557TmDFjTD2vv6AeV4167BvqcWXU46OjHh+/uqjHdkUDHAAANEgLFizQ+PHjNWnSJK1bt059+/bV4MGDlZGRYXU0yy1dulTjxo3Tl19+qfT0dJWVlSk1NVWFhYVWR7Od1atXa/bs2erSpYvVUWzjwIED6tOnj4KDg/Xxxx9rw4YNeuqpp9SoUSOro9nC1KlTNWvWLD3//PP66aef9Pjjj+uJJ57Qc889Z3U0R0tKSlJsbKxnS0tLq/K4yZMny+VyVbutWbPG6z67du3Seeedp2HDhmn06NH18XQaFOrx0VGPa456XBn1uHrUY/jCZRg2n6TFZBUrndp1VVIAAOqav9TCnj17qlu3bpo5c6ZnX8eOHXXxxRcftfHhVHv27FFCQoKWLl2qfv36WR3HNgoKCtStWzfNmDFDjzzyiE477TRNnz7d6liWu++++/TFF18wgvMoLrjgAiUmJmrOnDmefZdddpkiIiL06quvWpjM/irqz9nhwxXkCjHlnGVGiRYfWqDMzEyvmhYaGqrQ0NBKx+/du1d79+6t9pxt2rRRWFiYpCPN74EDB6pnz56aN2+eAgIYQ/Zn1OOaox5XjXpcNepx9ajHtVeX9diunzGp3gAAoMEpKSnR2rVrlZqa6rU/NTVVK1eutCiVfeXm5koSc9f+ybhx43T++efrL3/5i9VRbOWDDz5Qjx49NGzYMCUkJKhr16564YUXrI5lG2eddZY+++wz/fLLL5Kkb7/9VitWrNCQIUMsTtaAuA1zN0kxMTFeW1XNb0mKj49Xhw4dqt0qmt87d+7UgAED1K1bN7300ks0v6tAPfYN9bhq1OOqUY+rRz02QR3UY7sKsjoAAACAr/bu3avy8nIlJiZ67U9MTFR2drZFqezJMAxNmDBBZ511ljp37mx1HNt488039c0332j16tVWR7GdLVu2aObMmZowYYL+/ve/6+uvv9Ztt92m0NBQXXfddVbHs9y9996r3NxcdejQQYGBgSovL9ejjz6qq666yupoMNGuXbs0YMAAtWrVSk8++aT27Nnj+VqzZs0sTGYv1OOaox5XjXp8dNTj6lGP4Qsa4AAAoMFyuVxetw3DqLTP6W655RZ99913WrFihdVRbCMzM1O33367Pv30U89IT/zO7XarR48eeuyxxyRJXbt21Y8//qiZM2fygVtH5jt+7bXXNH/+fHXq1Enr16/X+PHj1aJFC11//fVWx2sYDEOSSYtl1dGMnp9++qk2bdqkTZs2qWXLln96SHuPcrMC9fjYqMeVUY+rRz2uHvXYBA2gHpuFBjgAAGhw4uPjFRgYWGl0WU5OTqVRaE5266236oMPPtCyZcsqNXCcbO3atcrJyVH37t09+8rLy7Vs2TI9//zzKi4uVmBgoIUJrdW8eXOdfPLJXvs6duyod955x6JE9nL33Xfrvvvu05VXXilJOuWUU7R9+3alpaXxgbuGDLchw2XOB+W6akaPHDlSI0eOrJNz+xPqcc1Qj6tGPa4e9bh61OPj1xDqsVmYxAwAADQ4ISEh6t69u9LT0732p6enq3fv3halsg/DMHTLLbfo3Xff1eLFi5WcnGx1JFsZNGiQvv/+e61fv96z9ejRQ9dcc43Wr1/v6A/bktSnTx9t3LjRa98vv/yi1q1bW5TIXoqKiirNBR0YGCi326QRVEADQj2uHvW4etTj6lGPq0c9hi8YAQ4AABqkCRMmaMSIEerRo4d69eql2bNnKyMjQ2PGjLE6muXGjRun+fPn6/3331d0dLRnZF5sbKzCw8MtTme96OjoSvOvRkZGKi4ujnlZJd1xxx3q3bu3HnvsMV1xxRX6+uuvNXv2bM2ePdvqaLYwdOhQPfroo2rVqpU6deqkdevWadq0abrxxhutjtZwGG6Zd8k1jQ6rUY+PjnpcPepx9ajH1aMem8BB9ZgGOAAAaJCGDx+uffv2acqUKcrKylLnzp310UcfMSpG0syZMyVJAwYM8Nr/0ksvcUk/jun000/XwoULNXHiRE2ZMkXJycmaPn26rrnmGquj2cJzzz2nBx54QGPHjlVOTo5atGihm2++WQ8++KDV0QBLUI+PjnqM40E9rh71GL5wGXafpMVkeXl5io2NVW5urmJiYqyOAwBAvaMWAgCsUFF/BrguUZAr2JRzlhmlWmIspKYBAFBDTqzHjAAHAAAAANQfB11yDQCAbTmoHjuuAV4x4D0vL8/iJAAAWKOiBjrsIjAAgE2UqVQyqQSVqdScEwEA4DBOqseOa4Dn5+dLkpKSkixOAgCAtfLz8xUbG2t1DACAQ4SEhKhZs2Zakf2Rqedt1qyZQkJCTD0nAAD+yon12HFzgLvdbu3atUvR0dFyuVymnDMvL09JSUnKzMy05Tw3/oTXuv7wWtcfXuv6w2t9hGEYys/PV4sWLRQQEGB1HACAgxw+fFglJSWmnjMkJERhYWGmnhMAAH/mtHrsuBHgAQEBatmyZZ2cOyYmxtENlfrEa11/eK3rD691/eG1FiO/AQCWCAsLs+2HYwAAnMJp9ZhhXwAAAAAAAAAAv0QDHAAAAAAAAADgl2iAmyA0NFQPPfSQQkNDrY7i93it6w+vdf3hta4/vNYAAAAAADiL4xbBBAAAAAAAAAA4AyPAAQAAAAAAAAB+iQY4AAAAAAAAAMAv0QAHAAAAAAAAAPglGuDHacaMGUpOTlZYWJi6d++u5cuXWx3J76Slpen0009XdHS0EhISdPHFF2vjxo1Wx3KEtLQ0uVwujR8/3uoofmnnzp269tprFRcXp4iICJ122mlau3at1bH8TllZme6//34lJycrPDxcbdu21ZQpU+R2u62OBgAAAAAA6hgN8OOwYMECjR8/XpMmTdK6devUt29fDR48WBkZGVZH8ytLly7VuHHj9OWXXyo9PV1lZWVKTU1VYWGh1dH82urVqzV79mx16dLF6ih+6cCBA+rTp4+Cg4P18ccfa8OGDXrqqafUqFEjq6P5nalTp2rWrFl6/vnn9dNPP+nxxx/XE088oeeee87qaAAAAAAAoI65DMMwrA7RUPXs2VPdunXTzJkzPfs6duyoiy++WGlpaRYm82979uxRQkKCli5dqn79+lkdxy8VFBSoW7f/b+9uY6qu3ziOfw4UAgpYgihGQjoVBAXBEjR1E3FilrMJEmHepDO1RKbmTTZLpXRJoAWFeTfSoOVNznUD3iLqQFA0ldKJclyiSDom2TQ45//AedYJ9E+lnjy9X9t5cK5z/c738/s9cbv88j29lJGRocWLFyskJERpaWm2jmVX5syZo/379/NXIw/Ac889J29vb61evdpSe/HFF+Xq6qrs7GwbJgMAAAAAAPcbO8D/pps3b6q0tFTR0dFW9ejoaB04cMBGqf4bamtrJUmPP/64jZPYr6lTp2rYsGGKioqydRS7tW3bNoWHh2vUqFFq27atQkNDtWrVKlvHskv9+vXTzp07derUKUnS0aNHVVhYqJiYGBsnAwAAAAAA99sjtg7wsKqpqVFDQ4O8vb2t6t7e3rp48aKNUtk/s9ms5ORk9evXT0FBQbaOY5dycnJ0+PBhHTp0yNZR7FpFRYUyMzOVnJysefPmqbi4WG+88YZatGihMWPG2DqeXXnzzTdVW1urbt26ydHRUQ0NDVqyZIni4+NtHQ0AAAAAANxnDMD/IYPBYPXebDY3quHemTZtmo4dO6bCwkJbR7FL58+f1/Tp05WXlydnZ2dbx7FrJpNJ4eHhSklJkSSFhobqxIkTyszMZAB+j+Xm5urzzz/Xxo0b1b17d5WVlSkpKUk+Pj565ZVXbB0PAAAAAADcRwzA/yZPT085Ojo22u1dXV3daFc47o3XX39d27ZtU0FBgZ544glbx7FLpaWlqq6uVlhYmKXW0NCggoICffTRR7px44YcHR1tmNB+tG/fXoGBgVa1gIAAbdq0yUaJ7NesWbM0Z84cjR49WpIUHBysyspKvffeewzAAQAAAACwc5wB/jc5OTkpLCxM+fn5VvX8/HxFRkbaKJV9MpvNmjZtmjZv3qxdu3bJ39/f1pHs1qBBg/TDDz+orKzM8goPD1dCQoLKysoYft9Dffv21U8//WRVO3XqlDp27GijRPbr+vXrcnCw/ufO0dFRJpPJRokAAAAAAMCDwg7wfyA5OVmJiYkKDw9XRESEsrKyZDQaNXnyZFtHsytTp07Vxo0b9fXXX8vNzc2y697Dw0MuLi42Tmdf3NzcGp2t3rJlS7Vp04Yz1++xGTNmKDIyUikpKYqNjVVxcbGysrKUlZVl62h2Z/jw4VqyZImefPJJde/eXUeOHFFqaqrGjx9v62gAAAAAAOA+M5jNZrOtQzzMMjIytGzZMlVVVSkoKEgffvih+vfvb+tYduVOZ6qvXbtWY8eOfbBh/oMGDhyokJAQpaWl2TqK3dm+fbvmzp2r06dPy9/fX8nJyZo4caKtY9mda9euacGCBdqyZYuqq6vl4+Oj+Ph4vf3223JycrJ1PAAAAAAAcB8xAAcAAAAAAAAA2CXOAAcAAAAAAAAA2CUG4AAAAAAAAAAAu8QAHAAAAAAAAABglxiAAwAAAAAAAADsEgNwAAAAAAAAAIBdYgAOAAAAAAAAALBLDMABAAAAAAAAAHaJATgAAAAAAAAAwC4xAAcAAAAAAAAA2CUG4ACazc/PT2lpabaOoQULFmjSpEm2jtGk3r17a/PmzbaOAQAAAAAAADEAB/6Vxo4dqxEjRljeDxw4UElJSQ9s/XXr1ql169aN6ocOHbL54PnSpUtKT0/XvHnzmn1NVVWVXnrpJXXt2lUODg53fJabNm1SYGCgWrRoocDAQG3ZsqVRT0ZGhvz9/eXs7KywsDDt27fP6vMFCxZozpw5MplMf+m+AAAAAAAAcO8xAAf+Q27evPmPrvfy8pKrq+s9SvP3rF69WhEREfLz82v2NTdu3JCXl5fmz5+vnj17Ntlz8OBBxcXFKTExUUePHlViYqJiY2NVVFRk6cnNzVVSUpLmz5+vI0eO6Nlnn9XQoUNlNBotPcOGDVNtba2+//77v32PAAAAAAAAuDcYgAP/cmPHjtXevXuVnp4ug8Egg8Ggc+fOSZJOnjypmJgYtWrVSt7e3kpMTFRNTY3l2oEDB2ratGlKTk6Wp6enBg8eLElKTU1VcHCwWrZsKV9fX02ZMkV1dXWSpD179mjcuHGqra21rLdw4UJJjY9AMRqNeuGFF9SqVSu5u7srNjZWly5dsny+cOFChYSEKDs7W35+fvLw8NDo0aN17do1S89XX32l4OBgubi4qE2bNoqKitKvv/56x+eRk5Oj559/3vL+8uXLateunVJSUiy1oqIiOTk5KS8vz5I7PT1dY8aMkYeHR5Pfm5aWpsGDB2vu3Lnq1q2b5s6dq0GDBlndb2pqqiZMmKBXX31VAQEBSktLk6+vrzIzMy09jo6OiomJ0RdffHHHewAAAAAAAMCDwQAc+JdLT09XRESEJk6cqKqqKlVVVcnX11dVVVUaMGCAQkJCVFJSou+++06XLl1SbGys1fXr16/XI488ov379+vTTz+VJDk4OGjFihU6fvy41q9fr127dmn27NmSpMjISKWlpcnd3d2y3syZMxvlMpvNGjFihK5cuaK9e/cqPz9fZ86cUVxcnFXfmTNntHXrVm3fvl3bt2/X3r179f7770u6dTRJfHy8xo8fr/Lycu3Zs0cjR46U2Wxu8llcvXpVx48fV3h4uKXm5eWlNWvWaOHChSopKVFdXZ1efvllTZkyRdHR0c1+zgcPHmzUP2TIEB04cEDSrd3zpaWljXqio6MtPbc9/fTTjY5GAQAAAAAAwIP3iK0DALg7Dw8POTk5ydXVVe3atbPUMzMz1atXL6udz2vWrJGvr69OnTqlLl26SJI6d+6sZcuWWX3nH8/A9vf316JFi/Taa68pIyNDTk5O8vDwkMFgsFrvz3bs2KFjx47p7Nmz8vX1lSRlZ2ere/fuOnTokHr37i1JMplMWrdundzc3CRJiYmJ2rlzp5YsWaKqqirV19dr5MiR6tixoyQpODj4jmtWVlbKbDbLx8fHqh4TE6OJEycqISFBvXv3lrOzs2XI3lwXL16Ut7e3Vc3b21sXL16UJNXU1KihoeGuPbd16NBBRqNRJpNJDg78PyMAAAAAAICtMJkBHlKlpaXavXu3WrVqZXl169ZN0q1d17f9cbf0bbt379bgwYPVoUMHubm5acyYMfrll1/uevTIn5WXl8vX19cy/JakwMBAtW7dWuXl5Zaan5+fZfgtSe3bt1d1dbUkqWfPnho0aJCCg4M1atQorVq1SlevXr3jmr/99pskydnZudFnH3zwgerr6/Xll19qw4YNTfb8PwaDweq92WxuVGtOj4uLi0wmk27cuPGXMwAAAAAAAODeYQAOPKRMJpOGDx+usrIyq9fp06fVv39/S1/Lli2trqusrFRMTIyCgoK0adMmlZaW6uOPP5Yk/f77781ev6nBb1P1Rx991Opzg8Egk8kk6dZ52fn5+fr2228VGBiolStXqmvXrjp79myTa3p6ekpSk0PyiooKXbhwQSaTSZWVlc2+j9vatWvXaCd3dXW1Zce3p6enHB0d79pz25UrV+Tq6ioXF5e/nAMAAAAAAAD3DgNw4CHg5OSkhoYGq1qvXr104sQJ+fn5qXPnzlavPw+9/6ikpET19fVavny5+vTpoy5duujChQv/d70/CwwMlNFo1Pnz5y21kydPqra2VgEBAc2+N4PBoL59++qdd97RkSNH5OTkpC1btjTZ26lTJ7m7u+vkyZNW9Zs3byohIUFxcXFavHixJkyYYPVjnM0RERGh/Px8q1peXp4iIyMl3XomYWFhjXry8/MtPbcdP35cvXr1+kvrAwAAAAAA4N5jAA48BPz8/FRUVKRz586ppqZGJpNJU6dO1ZUrVxQfH6/i4mJVVFQoLy9P48ePv+vwulOnTqqvr9fKlStVUVGh7OxsffLJJ43Wq6ur086dO1VTU6Pr1683+p6oqCj16NFDCQkJOnz4sIqLizVmzBgNGDCgyWNXmlJUVKSUlBSVlJTIaDRq8+bNunz58h0H6A4ODoqKilJhYaFVff78+aqtrdWKFSs0e/ZsBQQEaMKECVY9t3fI19XV6fLlyyorK7MapE+fPl15eXlaunSpfvzxRy1dulQ7duywOi89OTlZn332mdasWaPy8nLNmDFDRqNRkydPtlpr3759f+kHOAEAAAAAAHB/MAAHHgIzZ86Uo6OjAgMD5eXlJaPRKB8fH+3fv18NDQ0aMmSIgoKCNH36dHl4eNz1hxdDQkKUmpqqpUuXKigoSBs2bNB7771n1RMZGanJkycrLi5OXl5ejX5EU7q1c3vr1q167LHH1L9/f0VFRempp55Sbm5us+/L3d1dBQUFiomJUZcuXfTWW29p+fLlGjp06B2vmTRpknJycizHqOzZs0dpaWnKzs6Wu7u7HBwclJ2drcLCQmVmZlquCw0NVWhoqEpLS7Vx40aFhoYqJibG6p5zcnK0du1a9ejRQ+vWrVNubq6eeeYZS09cXJzS0tL07rvvKiQkRAUFBfrmm28sP+ApST///LMOHDigcePGNfs5AAAAAAAA4P4wmM1ms61DAEBzmc1m9enTR0lJSYqPj7d1nEZmzZql2tpaZWVl2ToKAAAAAADAfx47wAE8VAwGg7KyslRfX2/rKE1q27atFi1aZOsYAAAAAAAAEDvAAQAAAAAAAAB2ih3gAAAAAAAAAAC7xAAcAAAAAAAAAGCXGIADAAAAAAAAAOwSA3AAAAAAAAAAgF1iAA4AAAAAAAAAsEsMwAEAAAAAAAAAdokBOAAAAAAAAADALjEABwAAAAAAAADYJQbgAAAAAAAAAAC79D9hxFLvoFc6gwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1500x500 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "原始权重矩阵的一部分(5x5):\n",
      "[[ 1.9020243  -0.98042786  0.47197205  0.00751896  0.45111972]\n",
      " [-0.6989953  -0.7539018  -1.6730322   0.895078    0.72842824]\n",
      " [ 1.1006526   0.32565323 -0.13668585  0.4227683   0.05465927]\n",
      " [-0.88375205  0.60963565 -1.5299315   0.43329597 -1.7178423 ]\n",
      " [ 1.3465533  -0.571326    0.07685361 -1.1320914  -1.1538632 ]]\n",
      "\n",
      "LoRA近似后的矩阵的一部分(5x5):\n",
      "[[ 1.4178535  -0.10202433  0.97077304 -0.0949342   0.46077988]\n",
      " [-1.1337396   0.3354595  -0.7775882   0.9027459   0.12311662]\n",
      " [ 1.002753    0.09579059  0.6856753   0.4798254   0.6510572 ]\n",
      " [-0.72393465  0.1751302  -0.49631235  0.4491832   0.00296068]\n",
      " [ 0.8751194  -0.5518392   0.6017585  -1.6507436  -0.66215307]]\n",
      "\n",
      "误差矩阵的一部分(5x5):\n",
      "[[ 0.4841708  -0.87840354 -0.498801    0.10245315 -0.00966015]\n",
      " [ 0.4347443  -1.0893613  -0.895444   -0.0076679   0.60531163]\n",
      " [ 0.09789956  0.22986263 -0.8223612  -0.05705711 -0.59639794]\n",
      " [-0.1598174   0.43450546 -1.0336192  -0.01588723 -1.720803  ]\n",
      " [ 0.47143394 -0.01948684 -0.52490485  0.5186522  -0.49171013]]\n",
      "\n",
      "统计指标:\n",
      "最大误差: 1.733717\n",
      "平均误差: 0.537437\n",
      "误差标准差: 0.383309\n",
      "矩阵相似度(相关系数): 0.776545\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 1. 创建一个模拟的原始权重矩阵\n",
    "def create_original_matrix(d=512, k=512):\n",
    "    \"\"\"\n",
    "    创建一个模拟的原始权重矩阵\n",
    "    d: 输入维度\n",
    "    k: 输出维度\n",
    "    \"\"\"\n",
    "    original_weight = torch.randn(d, k)  # 随机初始化一个权重矩阵\n",
    "    return original_weight\n",
    "\n",
    "# 2. 实现LoRA分解\n",
    "class LoRALayer:\n",
    "    def __init__(self, d, k, r):\n",
    "        \"\"\"\n",
    "        d: 输入维度\n",
    "        k: 输出维度\n",
    "        r: LoRA秩 (rank)\n",
    "        \"\"\"\n",
    "        self.d = d\n",
    "        self.k = k\n",
    "        self.r = r\n",
    "        \n",
    "        # 初始化A和B矩阵\n",
    "        self.lora_A = torch.randn(d, r) / np.sqrt(r)  # 缩放初始化\n",
    "        self.lora_B = torch.zeros(r, k)  # B初始化为0\n",
    "        \n",
    "        # 使其需要梯度\n",
    "        self.lora_A.requires_grad_(True)\n",
    "        self.lora_B.requires_grad_(True)\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"前向传播\"\"\"\n",
    "        return (self.lora_A @ self.lora_B) @ x\n",
    "\n",
    "    def get_weight_update(self):\n",
    "        \"\"\"获取权重更新矩阵\"\"\"\n",
    "        return self.lora_A @ self.lora_B\n",
    "    \n",
    "# 3. 演示训练过程\n",
    "def train_lora(original_weight, lora_layer, num_iterations=1000):\n",
    "    \"\"\"\n",
    "    训练LoRA来近似原始权重矩阵\n",
    "    Adam是一种优化器，用于更新模型参数\n",
    "    [lora_layer.lora_A, lora_layer.lora_B]指定需要优化的参数\n",
    "    lr=0.01是学习率，控制每次更新的步长\n",
    "    优化器的作用是根据梯度更新A和B矩阵，使得BA的乘积逐渐接近原始权重矩阵\n",
    "    \"\"\"\n",
    "    optimizer = torch.optim.Adam([lora_layer.lora_A, lora_layer.lora_B], lr=0.01)\n",
    "    losses = []\n",
    "\n",
    "    for i in range(num_iterations):\n",
    "        # 计算当前LoRA权重\n",
    "        current_weight = lora_layer.get_weight_update()\n",
    "        \n",
    "        # 计算与原始权重的差异\n",
    "        # 计算当前LoRA权重与原始权重的差异\n",
    "        # 使用均方误差损失函数（MSE）来衡量差异\n",
    "        # 损失函数越小，表示当前LoRA权重越接近原始权重\n",
    "        loss = torch.nn.functional.mse_loss(current_weight, original_weight)\n",
    "        \n",
    "        # 反向传播\n",
    "        optimizer.zero_grad()  # 清除之前的梯度\n",
    "        loss.backward()       # 计算梯度\n",
    "        optimizer.step()      # 根据梯度更新参数\n",
    "        \n",
    "        if i % 100 == 0:\n",
    "            losses.append(loss.item())\n",
    "            print(f\"Iteration {i}, Loss: {loss.item():.6f}\")\n",
    "    \n",
    "    return losses\n",
    "\n",
    "\n",
    "# 4. 可视化结果\n",
    "# 修改可视化函数\n",
    "def visualize_results(original_weight, lora_approximation, losses):\n",
    "    plt.figure(figsize=(15, 5))\n",
    "    \n",
    "    # 绘制损失曲线\n",
    "    plt.subplot(131)\n",
    "    plt.plot(losses)\n",
    "    plt.title('Training Loss')\n",
    "    plt.xlabel('Iterations (x100)')\n",
    "    plt.ylabel('MSE Loss')\n",
    "    \n",
    "    # 绘制原始权重矩阵\n",
    "    plt.subplot(132)\n",
    "    plt.imshow(original_weight.detach().numpy(), cmap='viridis')\n",
    "    plt.title('Original Weight Matrix')\n",
    "    plt.colorbar()\n",
    "    \n",
    "    # 绘制LoRA近似后的矩阵\n",
    "    plt.subplot(133)\n",
    "    plt.imshow(lora_approximation.detach().numpy(), cmap='viridis')\n",
    "    plt.title('LoRA Approximation')\n",
    "    plt.colorbar()\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "    \n",
    "    # 打印矩阵数值\n",
    "    print(\"\\n原始权重矩阵的一部分(5x5):\")\n",
    "    print(original_weight.detach().numpy()[:5, :5])\n",
    "    \n",
    "    print(\"\\nLoRA近似后的矩阵的一部分(5x5):\")\n",
    "    print(lora_approximation.detach().numpy()[:5, :5])\n",
    "    \n",
    "    # 计算误差矩阵\n",
    "    error_matrix = original_weight.detach().numpy() - lora_approximation.detach().numpy()\n",
    "    print(\"\\n误差矩阵的一部分(5x5):\")\n",
    "    print(error_matrix[:5, :5])\n",
    "    \n",
    "    # 计算一些统计指标\n",
    "    print(\"\\n统计指标:\")\n",
    "    print(f\"最大误差: {np.abs(error_matrix).max():.6f}\")\n",
    "    print(f\"平均误差: {np.abs(error_matrix).mean():.6f}\")\n",
    "    print(f\"误差标准差: {np.abs(error_matrix).std():.6f}\")\n",
    "    \n",
    "    # 计算相似度\n",
    "    from scipy.stats import pearsonr\n",
    "    orig_flat = original_weight.detach().numpy().flatten()\n",
    "    lora_flat = lora_approximation.detach().numpy().flatten()\n",
    "    correlation, _ = pearsonr(orig_flat, lora_flat)\n",
    "    print(f\"矩阵相似度(相关系数): {correlation:.6f}\")\n",
    "    \n",
    "# 5. 主函数\n",
    "def main():\n",
    "    # 设置维度\n",
    "    d, k = 10, 10  # 使用较小的维度便于演示\n",
    "    r = 2  # LoRA秩\n",
    "    \n",
    "    # 创建原始权重\n",
    "    original_weight = create_original_matrix(d, k)\n",
    "    \n",
    "    # 创建LoRA层\n",
    "    lora_layer = LoRALayer(d, k, r)\n",
    "    \n",
    "    # 训练LoRA\n",
    "    losses = train_lora(original_weight, lora_layer)\n",
    "    \n",
    "    # 获取最终的LoRA近似\n",
    "    final_approximation = lora_layer.get_weight_update()\n",
    "    \n",
    "    # 计算参数量的节省\n",
    "    original_params = d * k\n",
    "    lora_params = r * (d + k)\n",
    "    reduction = (1 - lora_params/original_params) * 100\n",
    "    \n",
    "    print(f\"\\n原始参数量: {original_params}\")\n",
    "    print(f\"LoRA参数量: {lora_params}\")\n",
    "    print(f\"参数减少: {reduction:.2f}%\")\n",
    "    \n",
    "    # 可视化结果\n",
    "    visualize_results(original_weight, final_approximation, losses)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74f7ef8",
   "metadata": {},
   "source": [
    "<div align=center><img src=\"https://typora-photo1220.oss-cn-beijing.aliyuncs.com/DataAnalysis/muyan/image-20250311162917012.png\" width=100%></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07baa042",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
